Context Observability

The Story

A contract management platform serves 12,000 enterprise users across legal, procurement, and finance teams. The product uses Claude Sonnet to analyze uploaded contracts, flag risky clauses, and suggest negotiation alternatives. Quality has been strong for six months -- user satisfaction at 91%, clause detection accuracy at 93%.

Then, over two weeks, clause detection accuracy drops to 81%. Legal teams start catching missed non-compete clauses and incorrectly flagged indemnification provisions. The support team gets escalations. The VP of Product asks: what happened?

The engineering team checks their logs. They can see every user query and every model response. They can tell you the accuracy dropped. They can't tell you why.

Did the model regress? Same Sonnet version. Did the prompts change? Untouched in four weeks. Did retrieval quality degrade? "We don't log what was retrieved." Did the memory system start loading wrong contract histories? "We log that memory was loaded, but not what was in it."

Three weeks ago, the document ingestion pipeline changed its chunking strategy -- from overlapping 512-token chunks to non-overlapping 1024-token chunks -- to reduce storage costs. The change was deployed to the ingestion pipeline, not the AI product, so no one on the product team was notified. The larger chunks reduced retrieval precision because the embedding model produced lower-quality vectors for 1024-token chunks.

The team spent eleven days diagnosing this. With context observability -- tracing what was retrieved, what scores the reranker assigned, what the model actually saw -- they would have seen the retrieval quality degradation within hours.

The Core Idea

Context observability is the practice of logging, tracing, and visualizing the complete context assembled for every inference call -- not just what the user said and what the model replied, but everything in between.

The mental model shift: traditional software has deterministic paths. Given the same input, you get the same output. AI products have a probabilistic path between input and output, AND that path runs through a context assembly pipeline that itself can have bugs, regressions, and failures. You have two debugging surfaces: the assembly pipeline (deterministic, inspectable) and the model inference (probabilistic, harder to inspect). Context observability gives you full visibility into the first.

Think of it like a flight data recorder on an aircraft. After an incident, investigators don't just know "the plane deviated from course." They know every instrument reading, every control input, every system state change, second by second. Without the black box, they have the departure gate and the crash site -- input and output -- and years of speculation about what happened in between.

The observability platform landscape. Three tools represent different approaches: Arize Phoenix (open-source, strong on retrieval analysis and embedding drift), LangSmith (pipeline visualization, dataset building from traces), and Braintrust (eval-first, where observability and evaluation share the same data). Provider-native tracing is also maturing -- Anthropic builds tracing directly into Claude Code, and OpenAI's Agents SDK includes structured span logging.

What most teams get wrong: logging output quality without logging input quality. Teams build dashboards that track response accuracy and user ratings. These tell you THAT quality changed. They don't tell you WHY. Context observability focuses on input quality: did retrieval return the right documents? Did memory load the right history? Did tools get the right selection? If the output is wrong AND the input was correct, it's a model problem. If the output is wrong AND the input was wrong, it's a harness problem.

Where This Hits in Production

Retrieval drift detection. In production, retrieval quality degrades silently. Knowledge bases grow. New documents dilute the embedding space. Chunking strategies change. Without observability on the retrieval stage -- tracking average retrieval scores, candidate counts, reranker lift over time -- you won't notice until users complain.

Multi-tenant debugging. In B2B products, quality problems are often tenant-specific. Tenant A's knowledge base has poorly formatted documents. Tenant B's configuration points to a deprecated tool. Span-level tracing with tenant tags lets you filter traces by tenant and diagnose tenant-specific issues without sifting through millions of global traces.

Compliance and audit trails. In regulated industries, organizations need to demonstrate what information the AI system used when making a recommendation. Context observability produces the audit trail: the exact context window the model received for any specific query.

In Practice: Anatomy of a Production Trace

Here's a trace for a single query in a contract management platform -- showing what was retrieved, what was assembled, what the model saw, with timestamps and token counts at every stage.

Remember This

1. If you can't see what the model saw, you can't debug what the model did. Context observability traces the 95% of the context window that isn't the user's message -- because that 95% determines the response.

2. Same tools, different lens: observability platforms serve both engineering debugging and product evaluation. Building two separate systems wastes money and loses the integration between input quality and output quality.

3. Span-level tracing across CONTEXT layers -- one tagged span per assembly stage -- is the production pattern. It turns "the AI was wrong" into "retrieval returned 3 stale documents because the index sync failed at 3am."