The Number You See Is Not What You Get
When Anthropic announced Claude’s 200,000-token context window, or when Google unveiled Gemini 1.5 Pro with a million-token window, the coverage treated it as straightforward progress. More tokens in, more capability out. The framing makes intuitive sense: if a model can see more text at once, it should be able to reason about more text at once.
This is not quite right. Context window size and context window effectiveness are two different things, and the gap between them is large enough to matter for anyone building or using these systems seriously.
A token is roughly three to four characters of English text, so 200,000 tokens is somewhere around 150,000 words, or about two average novels. Fitting two novels into a context window is a genuine engineering achievement. What the model does with those novels is governed by architecture and training choices that the headline number says nothing about.
Why the Window Feels Smaller Than Advertised
The core mechanism of transformer-based language models is self-attention. Every token attends to every other token, and those attention weights determine what information flows where. The computational cost of this scales quadratically with sequence length, which is why long context windows require significant engineering to even function. But the more important issue for users is that attention is not uniform.
Research on this problem has produced a consistent finding: models tend to perform well on information at the beginning and end of long contexts, and poorly on information in the middle. This is sometimes called the “lost in the middle” problem, after a 2023 paper from Stanford and other researchers that demonstrated the effect empirically. They found that when the answer to a multi-document question was placed in the middle of a long context, retrieval accuracy dropped significantly compared to when the answer appeared near either end.
In practice, this means if you paste a 100-page document into a model and ask a question whose answer is on page 47, you should expect worse performance than if the answer were on page 3 or page 99. The model isn’t lying when it fails to find that information. It genuinely has degraded access to it.
The Needle-in-a-Haystack Test and Its Limits
The standard benchmark for evaluating long-context capability is called the “needle in a haystack” test. You hide a specific piece of information (the needle) somewhere in a long document (the haystack) and ask the model to retrieve it. Some models score quite well here. GPT-4 Turbo achieved high accuracy on this benchmark with contexts up to 128,000 tokens.
Here is the problem with this benchmark: real tasks are not needle-in-a-haystack tasks. Finding a single planted fact is a retrieval problem. Reasoning across an entire long document, synthesizing themes, noticing contradictions, tracking how arguments develop over pages, those are comprehension problems. They require the model to integrate distributed information, not locate a specific string.
Many models that perform well on needle-in-a-haystack benchmarks show substantially weaker performance on tasks requiring genuine multi-hop reasoning across long contexts. The benchmark measures one thing. The headline context window number measures something else. Neither measures what most people actually want from long-context models.
It’s a version of Goodhart’s Law applied to AI evaluation: when a measure becomes a target, it stops being a reliable measure of the underlying capability you care about.
Why the Window Is Also Bigger Than It Sounds
Here is where it gets interesting, because the story cuts both ways.
The naive mental model of a context window is a finite reading buffer. You put things in, you use them up, the model forgets them when they scroll out. This makes the window sound like a hard constraint, a container with a fixed volume.
But for knowledge-intensive tasks, the context window interacts with what the model already knows from pretraining in ways that aren’t obvious. A model asked to reason about a code library it was trained on extensively will perform differently than a model reasoning about a document it has never seen before, even if both fit in the same context window. The in-context information gets interpreted through the lens of existing model knowledge.
This means that in some scenarios, you are effectively getting more reasoning capability than the raw context capacity would suggest, because the model is supplementing context with parametric memory. Ask Claude to review a Python script that uses a well-known framework, and it isn’t reasoning purely from the code you pasted. It’s drawing on patterns from its training to fill in implied behavior, known library behaviors, common bugs, and so on.
The limitation flips when you’re working with genuinely novel or specialized information. If you’re feeding a model proprietary internal documents it has never encountered, there’s no training knowledge to supplement context. The model has only what you gave it, and the degradation effects in the middle of long documents hit harder.
How Retrieval Augmentation Changes the Picture
Most production applications that need to process large document sets don’t dump everything into a context window. They use retrieval-augmented generation (RAG): a system that chunks documents into smaller segments, embeds them as vectors, and retrieves only the most relevant segments before passing them to the model.
RAG sidesteps the lost-in-the-middle problem by ensuring the model sees fewer, more relevant chunks rather than one enormous document. It’s architecturally elegant and, for many use cases, produces better results than brute-force long context even when long context is technically available.
However, RAG introduces its own failure modes. Retrieval is semantic, not perfect. A retriever might miss the relevant chunk because the query doesn’t overlap well with the chunk’s phrasing. Multi-hop questions, where answering question A requires first knowing B, which requires finding C, are especially hard because a single retrieval step can’t handle the dependency chain.
The industry is settling on a pragmatic answer: use both. Long context windows are genuinely useful for tasks where you want the model to have holistic awareness of a document, dense code files being a good example. RAG is better when you’re querying across a large corpus of documents with diverse coverage. Knowing which to reach for is a practical skill that has nothing to do with the headline context number.
The Quadratic Cost Problem Has Not Gone Away
For developers and teams deciding how to architect LLM applications, the compute cost of long contexts matters. Attention’s quadratic scaling with sequence length means that doubling your context length roughly quadruples the compute for the attention computation. This is why inference costs for long-context models are significantly higher per query, and why providers price long-context usage at a premium.
There are architectural innovations designed to address this. Sparse attention mechanisms, linear attention variants, and techniques like sliding window attention all try to approximate full attention at lower computational cost. Anthropic, Google, and others have their own proprietary approaches. But the gains are partial. Nobody has made the scaling problem disappear, they’ve made it more manageable.
For anyone building on these APIs, this creates a real economic tension. You can afford to stuff large contexts for occasional high-value queries. You cannot afford to do it on every request in a high-volume system. The cost pressure shapes how context windows get used in practice, which is often more conservatively than the advertised limits would suggest.
What This Means
Context windows are a real capability, not marketing fiction. The jump from 4,000 tokens to 200,000 tokens represents genuine engineering progress and opens up tasks that were previously impossible: reviewing an entire codebase in a single pass, processing lengthy contracts, keeping long conversation histories intact.
But the number is not a performance guarantee. Attention is uneven across long sequences, with known degradation in the middle. Benchmark performance on retrieval tasks doesn’t reliably predict performance on reasoning tasks. Models supplementing context with training knowledge can outperform their context limits on familiar domains while underperforming on unfamiliar ones.
The useful mental model is this: a context window is like a whiteboard. The size of the whiteboard matters. But a large whiteboard doesn’t automatically make you better at working through a hard problem. What you write on it, how you organize it, and what you already know before you walk into the room all shape the outcome. The people getting the most out of long-context models treat context curation as a first-class design problem, not an afterthought.
If you’re designing prompts or systems, read about how your prompt changes before the model ever processes it. Context length is one variable. It’s not the only one that matters.
