Updated June 2026

The spec sheet says the model handles a million tokens. The spec sheet is telling the truth about capacity and nothing about quality. What matters in practice is how attention behaves as the window fills, and it degrades long before the hard limit.

It’s a dilution, not a cliff

Quality doesn’t fall off a cliff at the context limit; it erodes gradually as the window fills. Every token competes for the model’s attention, so the more there is, the less each gets. A question that gets a sharp answer against 5K tokens of relevant context gets a mushier one against 200K tokens where the relevant part is buried in transcript sediment. The model doesn’t fail loudly; it just gets slightly worse at noticing things, and slightly worse compounds.

The related effect, historically strong and still present to a degree even as recall benchmarks improve: material in the middle of a long context is recalled worse than material at the start or end. If you’re stuffing documents into a prompt, position matters. Put the most load-bearing content where attention is strongest, and put the question or instruction near it.

More context is not more better

The instinct is to give the model everything, just in case. Resist it. Irrelevant context isn’t neutral; it’s noise the model must read, pay for, and discount, and it provides surface area for the model to latch onto the wrong thing. Curating five relevant files beats dumping fifty, both on cost and on answer quality. Retrieval, sub-agents, and plain old grep exist so the main context can hold conclusions instead of raw material.

Long sessions accumulate sediment

In agent sessions specifically, the transcript fills with things that were useful once and never again: the full text of files that have since been edited, tool outputs from dead-end explorations, three rounds of debugging a problem that’s now fixed. The agent re-reads all of it on every turn. This is why long sessions get expensive (covered in Managing Token Costs) and also why they get dumber: the signal-to-sediment ratio falls.

What helps:

  1. Start fresh more often than feels natural. A new session with a crisp summary of where things stand routinely outperforms hour ten of an old one. The summary forces curation; the old transcript prevents it.
  2. Externalize state. Notes files, TODO lists, and design docs on disk survive a fresh start and cost nothing per turn. Context is working memory, not storage.
  3. Use compaction and pruning where available. Server-side compaction summarizes the old transcript; context editing drops stale tool results. Both trade perfect recall of the past for sharpness in the present, which is usually the right trade.
  4. Watch for the symptoms. When an agent starts forgetting constraints from earlier, repeating work, or contradicting its own decisions, that’s context degradation, not a model having a bad day. The fix is hygiene, not prompting harder.

The mental model: the context window is a desk, not a filing cabinet. Big desks are nice, but the work goes better when the desk holds only the current job.