Context Window

What the context window is

The context window (also called context length) is the maximum number of tokens an LLM can attend to in a single forward pass. Input tokens (your prompt + conversation history) and output tokens (the generated response) together must fit within this limit.

Tokens are roughly 3/4 of a word in English: "hello world" = 2 tokens, a typical paragraph = 100–150 tokens, a typical A4 page = 400–600 tokens. Common benchmarks: 4K tokens ≈ 3,000 words, 128K tokens ≈ 96,000 words ≈ a short novel, 1M tokens ≈ 750,000 words ≈ the full Harry Potter series.

The context window is not just about length — it's about what the model can reason over. Everything in the context is attended to simultaneously (O(n²) attention), and positions early in the context can affect positions late in the context. This is fundamentally different from a database query or search index.

KV-cache and memory cost

During autoregressive generation, the model computes key and value tensors for every token in the context at every layer. These are cached to avoid recomputation — the KV-cache. Memory cost scales as:

KV-cache bytes = 2 × num_layers × num_kv_heads × head_dim × context_length × bytes_per_element

For Llama 3.1 70B (fp16): 80 layers × 8 KV heads × 128 head_dim × 2 bytes × 2 (K+V) = 328 bytes per token. At 128K tokens: 328 × 128,000 = 42GB of KV-cache alone — more than the model weights.

def kv_cache_gb(num_layers, num_kv_heads, head_dim, seq_len, dtype_bytes=2):
    # Returns KV-cache memory in GB
    kv_per_token = 2 * num_layers * num_kv_heads * head_dim * dtype_bytes
    return kv_per_token * seq_len / 1e9

# Llama 3.1 70B
print(f"128K context: {kv_cache_gb(80, 8, 128, 128_000):.1f} GB")
print(f"8K context:   {kv_cache_gb(80, 8, 128, 8_000):.1f} GB")
# 128K context: 41.9 GB
# 8K context:    2.6 GB

This is why serving infrastructure matters: long context requests are memory-hungry and require careful batching to avoid OOM.

Position encoding limits

LLMs learn position information through positional encodings (RoPE, ALiBi, or absolute). These encodings are trained on sequences up to the training context length. At inference time, extrapolating beyond the training length can degrade quality.

RoPE (Rotary Position Embedding): Used by Llama, Mistral. Can be extrapolated beyond training length using scaling techniques (YaRN, LongRoPE). Models like Llama 3.1 use YaRN scaling to extend 8K training length to 128K inference length.

ALiBi: Uses attention biases that scale linearly with distance. Better extrapolation than learned absolute positions, but still degrades for very long contexts.

The practical implication: a model advertised as "128K context" may actually extrapolate a shorter training context. Check whether quality benchmarks at long context are included in the model card.

Lost in the middle

A surprising finding (Liu et al. 2023): LLMs perform best when relevant information is at the start or end of the context, and worst when it's in the middle. Performance degrades roughly in a U-shape across position.

# Demonstrate "lost in the middle" with a simple test
import openai

def ask_with_position(relevant_doc, irrelevant_docs_before, irrelevant_docs_after, question):
    context = "

".join(
        irrelevant_docs_before + [relevant_doc] + irrelevant_docs_after
    )
    return client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role":"user","content":f"{context}

Q: {question}"}],
    ).choices[0].message.content

# Quality drops when relevant_doc is buried in the middle:
# Position 0/10 docs: ~90% accuracy
# Position 5/10 docs: ~65% accuracy (middle = worst)
# Position 10/10 docs: ~85% accuracy

Mitigations: put the most important context at the start or end, use retrieval to return only the most relevant chunks, or re-rank retrieved chunks to place the best ones at the boundaries.

Long context strategies

Given the cost and quality trade-offs, here's when to use different approaches:

Use long context directly when: you need to reason over the relationships between many parts of a document (not just retrieve a fact), the document is <50K tokens (cost manageable), or you're prototyping and don't want to build a retrieval pipeline.

Use RAG instead when: the total corpus is too large to fit in context, you need to search across many documents, cost is a concern (>50K tokens per query gets expensive), or you need to cite specific sources.

Hybrid approach: retrieve the top-K relevant chunks, then pass them in the context. This combines the scalability of retrieval with the reasoning power of in-context processing.

Chunk-and-summarise: For very long documents, recursively summarise chunks and query the summaries. Loses detail but scales to arbitrarily long inputs.

Model comparison

Gotchas

Input vs output tokens: Context windows include both input and output. If your context is 100K tokens, you have little room left for output. Always reserve capacity: if context window = 128K and input = 120K, max output = 8K.

Counting tokens: Use the model's tokenizer to count before sending. A request that exceeds the context limit will fail with an API error. For OpenAI models, use the tiktoken library. For others, use the HF tokenizer.

import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
token_count = len(enc.encode(your_text))
print(f"Token count: {token_count}")

Billing: You pay for the full context on every API call. With a 50K-token system prompt and 128K context models, sending one message costs you 50K input tokens even if the user typed two words.