The Intuition That Leads You Wrong
When Anthropic announced Claude’s 100K token context window, and then Google pushed Gemini to 1 million tokens, the dominant reaction in developer communities was relief. Finally, you could just dump everything in. Paste the whole codebase. Feed it the entire research paper. Stop worrying about what to include and just include everything.
This intuition is wrong, and the wrongness has a specific shape that’s worth understanding before you build something that depends on it.
The problem is that context windows and memory are not the same thing. A context window is a fixed-size buffer of tokens that an LLM can attend to during inference. Memory implies that information is reliably accessible and retrievable. These overlap sometimes, but they diverge in ways that matter practically, and the longer the context, the wider that divergence gets.
What Attention Actually Does
To understand why longer contexts behave worse, you need a rough model of how transformers work. The key mechanism is self-attention, which lets the model relate any token to any other token in the context. When the model is processing a token, attention weights determine how much it “looks at” each other token. High weight means that token is pulling strongly on the output. Low weight means it’s barely registering.
Think of it like a spotlight system on a stage with hundreds of actors. The model can point spotlights at any of them, but the total amount of light is fixed. As you add more actors, each one gets dimmer on average. Some will be almost invisible.
The question is: which actors get ignored? Researchers have found a consistent pattern. Information at the very beginning of a context (primacy) and information at the very end (recency) gets reliably attended to. Information buried in the middle gets systematically underweighted. This has been empirically demonstrated in multiple studies, most notably the “Lost in the Middle” paper from Liu et al. (2023), which tested how well models could retrieve information from documents placed at different positions in long contexts. Performance degraded sharply when the relevant information was placed in the middle, even when the total context was well within the model’s stated limit.
The Needle-in-a-Haystack Problem Is Not Solved
The benchmark that context window announcements usually cite is called the “needle in a haystack” test. You hide a specific fact somewhere in a long document and ask the model to retrieve it. Recent large-context models do reasonably well on this benchmark, which is why it gets cited.
But retrieval of a planted fact is the easiest possible case. The fact is explicit, unambiguous, and the question is designed to point directly at it. Real-world usage looks nothing like this.
In practice, you’re asking the model to synthesize, reason across, and weigh multiple pieces of information distributed throughout a long context. You’re not asking “what is the API key mentioned on page 47.” You’re asking “given everything in this codebase, what are the likely failure modes of this new function I’m adding?” That requires the model to hold a coherent understanding of the entire context, not just retrieve one buried sentence.
For synthesis tasks, the degradation at long contexts is significantly worse than retrieval benchmarks suggest. The model doesn’t just miss facts. It develops subtly incorrect understandings of the material, and because LLM confidence doesn’t track accuracy, it will state those incorrect understandings with the same fluency it uses for correct ones.
Why This Gets Worse as Windows Grow
Here’s the compounding problem: as context windows grow, the cost of the failure mode grows with them.
With a 4K token context (roughly 3,000 words), you’re forced to be selective. You have to think about what matters and include only that. This constraint is annoying but it’s also a forcing function that tends to produce better results, because the information you include is dense with signal and the model can attend to all of it reasonably well.
With a 100K token context (roughly 75,000 words), you stop being selective. You include everything because you can. This feels efficient but it means you’re now feeding the model a massive amount of noise alongside the signal, and asking it to figure out which is which. The model is doing the curation work you offloaded onto it, and it’s doing that work less reliably than you would.
With a 1 million token context, you’re feeding in entire repositories, documentation sets, or datasets. The middle of that context might be thousands of pages of material that the attention mechanism is systematically underweighting. You’ve given the model more information and less useful information at the same time.
The vendors aren’t lying about the token counts. The windows are real. What’s not real is the implied claim that every token in that window is being considered with equal fidelity.
What Actually Works Better
Retrieval-Augmented Generation (RAG) exists specifically to address this problem, and it’s worth understanding why it works even though it sounds like a step backward.
In a RAG setup, instead of stuffing everything into the context, you maintain an external index (usually a vector database storing embeddings of document chunks). When a query comes in, you retrieve the most relevant chunks based on semantic similarity and inject only those into the context. The model works with a small, highly relevant context rather than a massive unfocused one.
This sounds worse because you might miss something. But the evidence suggests you miss less than you would with naive full-context injection. A context window with 3,000 highly relevant tokens consistently outperforms one with 100,000 loosely relevant tokens for complex reasoning tasks. You’re trading coverage for coherence, and coherence usually wins.
The other approach that works is explicit chunking with summarization. Process long documents in sections, build intermediate summaries, then reason across the summaries. This is closer to how humans actually read and synthesize long material. We don’t hold 75,000 words in working memory. We build hierarchical abstractions.
For code specifically, which is where developers most often want to use huge contexts, targeted retrieval of relevant functions and modules tends to produce better results than pasting the entire codebase. The model needs the interface of a function you’re calling, not the entire history of every file in the repository.
The Benchmark Problem
Part of why this is counterintuitive is that the benchmarks used to market context window improvements don’t measure what most developers actually need.
A model that scores well on “retrieve a specific fact from a 500K token document” is not necessarily a model that reasons well across a 500K token document. These are fundamentally different cognitive tasks. The first is closer to keyword search. The second requires integrated understanding.
When evaluating models for your specific use case, you need to test the actual task, not a retrieval proxy for it. Put your representative inputs at different positions in the context. Test with your actual queries. Measure output quality directly. You’ll often find that a model with a smaller effective window but better reasoning behavior outperforms a model with a larger stated limit.
This connects to a broader issue with how LLMs handle unfamiliar or sparse context: they don’t fail noisily. They fail confidently, producing plausible-sounding outputs that are subtly wrong about the material you gave them.
What This Means for How You Build
The practical takeaway is that context window size should be treated as a ceiling, not a target. The fact that you can put 1 million tokens in doesn’t mean you should. For most tasks, the optimal context is the smallest context that contains the information needed to answer the question well.
This means:
Be intentional about what you include. Don’t paste in whole files when a relevant function or section would do. Don’t include documentation that isn’t related to the specific task. The model is not better at ignoring irrelevant information in larger contexts. It’s worse.
Position matters more than you think. If you have critical information, put it at the start or end of the context. Information in the middle of a very long prompt is at real risk of being underweighted. This is not a speculation. It’s documented behavior.
Use retrieval architectures for large knowledge bases. If your application needs to reason over a large corpus, RAG is not a workaround. It’s the right architecture. Larger context windows don’t obsolete it.
Test with long contexts explicitly. If you’re using long contexts in production, your test cases need to specifically verify that information from different positions in the context is being correctly used. Middle-of-context failures won’t show up in simple happy-path testing.
The vendors will keep pushing context windows larger, and they’ll keep presenting that growth as unambiguous progress. In some ways it is. The ceiling rising is genuinely useful. But the ceiling is not the floor, and treating it as one will produce systems that are quietly, confidently wrong in ways that are hard to catch until they matter.
