The Context Window

The Story

Filling a 128,000-token context window past 90,000 tokens made an insurance company's claims assistant worse, not better — and the engineering team's first diagnosis was the wrong one. The product pulled policy documents, claim histories, and regulatory guidelines into the model's context window — the maximum amount of text the model can hold in mind at once — for every adjuster query. The team had picked the largest window they could find, on the assumption that more room meant more information meant better answers.

(Tokens — chunks of text the model processes, usually a few letters or a short word. A page of normal English is roughly 500–700 tokens. 128,000 tokens is roughly a 200-page book.)

The system works brilliantly on simple claims. A straightforward auto accident with one policy and a clear coverage section. The adjusters love it. They praise the speed and the precision of the citations.

Then adjusters start working complex claims. A commercial property loss with multiple coverage endorsements, three years of correspondence, building inspection reports, and overlapping state regulations. The context fills to 90,000 tokens. The system has all the information.

The answers get worse.

The AI cites the wrong endorsement. It mixes up coverage limits from one section with deductibles from another. It misses exclusion clauses buried in the middle of long documents. The adjusters — who had been trusting the precise citations — file several claims with incorrect coverage assessments before anyone spots the pattern.

The engineering team's first diagnosis: the model isn't smart enough for complex claims. That diagnosis is wrong. The team filled the context window past the point where the model could actually use what was in it. The information was there. The model's ability to find and weigh it had quietly collapsed.

This pattern is everywhere in production AI in 2026. The fix is not a smarter model. The fix is a smaller, sharper context.

The Core Idea

The context window is the most misunderstood concept in AI product design. Teams treat it like a bucket — pour information in until it's full, then ship it. It does not work like a bucket. It works like a spotlight.

When an LLM processes the input you send it, it uses a mechanism called attention. Attention is the math that decides how much weight each piece of text gets when the model generates its answer — like a reader's eyes moving across a page, lingering on some parts, skimming others. The mechanism does not weigh every chunk of text equally.

Information at the beginning of the context gets strong attention. It's the model's first impression. Information at the end gets strong attention. It's what the model saw most recently. Information in the middle gets diluted.

This is not a small effect. A landmark research paper from Stanford and UC Berkeley — Lost in the Middle: How Language Models Use Long Contexts — tested this systematically. The researchers placed correct answers at different positions inside a long context and measured how often models found them. Models performed best when the relevant information appeared at the very beginning or the very end. When it appeared in the middle, accuracy dropped by 20 percentage points or more.

Source: Liu et al., Lost in the Middle, NeurIPS 2023.

Think of it like: reading a 300-page report in one sitting. You remember the executive summary clearly. You remember the conclusion clearly. Pages 120 to 180? A blur. Same report. Same reader. The middle dissolves.

The attention degradation problem has a sharper edge than most teams realise. Chroma — a company that builds vector databases for AI applications — published research on what they called context degradation. Their finding: model performance does not degrade smoothly as you add tokens. It holds steady up to a threshold — roughly 50–60% of the model's maximum context window — and then drops more steeply. Chroma named this the pre-rot threshold: the point where context starts actively working against you.

Anthropic's engineering team confirmed a related finding for long-running applications. For tasks that pile up context over time — like extended coding sessions or multi-step agent workflows — resetting the context with carefully selected information outperforms compressing the accumulated context. Less, well-chosen, beats more.

A model with a 200,000-token window does not mean your product benefits from 200,000 tokens of context. For most production use cases, the sweet spot is dramatically smaller — typically 8,000 to 32,000 tokens of carefully selected, well-positioned information. The rest of the window is capacity you could use, not capacity you should use.

Position matters. Critical instructions and constraints belong at the beginning of the context — the system prompt position. The user's current question and the most relevant retrieved information belong near the end. Supporting but less critical information fills the middle, where attention is weakest.

Density matters. A context with 10,000 tokens of precisely relevant information beats a context with 50,000 tokens where the relevant information is buried in noise. Every irrelevant token competes for attention against the relevant ones.

Volume has diminishing returns. The first 4,000 tokens of well-chosen context have an enormous impact on response quality. By the time you're loading the fifth or sixth set of documents, each additional chunk is contributing less and increasingly confusing the model more than helping it.

Where This Hits in Production

Multi-tenant context variance. In B2B platforms, different customers have different document sizes. Customer A's policy manual is 2,000 tokens. Customer B's is 18,000 tokens. Customer B's query pushes past the pre-rot threshold while Customer A's stays safely below it. The PM has to design around this variance — chunking large documents or dynamically adjusting how much supporting context loads.

Cost at scale. Context window usage drives inference cost directly. At $3 per million input tokens, a product loading 20,000 tokens per query at 500,000 queries a day spends $30,000 a day on input alone. Achieving the same quality with 8,000 tokens drops the daily cost to $12,000. That is $6.5 million per year saved from context optimisation alone.

Latency implications. Larger context means longer processing time. A query with 50,000 input tokens measurably takes longer than one with 8,000. For real-time applications — customer support, coding assistants, search — that latency difference is something users feel.

Connecting the Dots

The context window is the bridge between Prompt ≠ Context (Agentic Stack L1-T01) and everything that follows. Every later topic — the seven-layer CONTEXT framework, context rot, memory architecture, tool design — is ultimately about managing what goes into this window and where it sits.

The context window is to AI products what working memory is to human cognition. Cognitive psychology has known for decades that human working memory holds roughly four to seven items at a time — Miller's magical number seven. The items at the beginning (primacy effect) and the end (recency effect) get remembered best. Sound familiar? The attention pattern in modern LLMs mirrors human primacy and recency. Something has to lose.

On the autonomy spectrum. Context window management gets harder at every level of autonomy. At Level 6 (autonomous agent), the agent itself is making the context assembly decisions in real time. The full spectrum lives in Spectrum of Agency (L3-T22).

The moat. A team using 32,000 tokens carefully on Claude Sonnet often outperforms a team stuffing 100,000 tokens into a frontier model — because attention quality beats attention quantity.

Remember This

1. The context window is not a bucket you fill. It is an attention field that degrades. Information at the beginning and end gets strong attention. The middle dissolves.

2. Performance peaks at roughly 50–60% of maximum capacity, then declines. Your product's context budget should target that sweet spot — not the maximum.

3. 8,000 tokens of precisely relevant, well-positioned context beats 50,000 tokens of comprehensive-but-noisy context. Density and position matter more than volume.

In Practice — The Context Budget

You're the PM for a commercial insurance claims assistant. Your model supports 200,000 tokens. The operating budget is 24,000 tokens — 12% of capacity. Empirical testing showed quality on complex claims peaked in the 20K–30K range and declined beyond 40K.

Knowledge gets the largest allocation (8,000 tokens) because the claim-specific information is what the adjuster actually needs. But it is capped at the primary policy plus the top three most relevant endorsements — not every endorsement on file. The retrieval system has to rank and select.

The buffer (4,500 tokens) exists for edge cases. It is a deliberate reserve. Not unclaimed space that gradually fills with noise.

The four numbers that belong on every AI PM's wall: average tokens per query, P95, P99, and quality score broken down by context size band. Without those four numbers, the team is flying blind on the most expensive, most quality-sensitive resource in the entire AI stack.

(P95 and P99 — the size of the query at the 95th and 99th percentile. Average lies. Tail percentiles tell the truth.)

Up Next — The Seven Layers That Fight for the Budget

The next question: what fights for space inside the window? Seven distinct kinds of information, each with its own purpose and its own owner. I call it the CONTEXT stack — Constitution, Observations, kNowledge, Tracks, Equipment, eXecution, Template. Each letter is a layer. Each layer is a product decision. The next post is The CONTEXT Stack (L1-T03).