Using Traces to Debug What Went Wrong

The Story

Consider a customer support bot with a classification step that routes requests to different handlers. A user asks, "Where is my order?" and gets a polished response about shipping policies from the FAQ handler. The team's first instinct: the model isn't understanding user intent. Fix: rewrite the system prompt.

Two engineers spend a day refining the prompt. The problem persists. A third engineer opens Phoenix and clicks into the failing request. The trace shows it immediately: the classification span routed "where is my order" to the "faq" category with high confidence. The model wasn't misunderstanding intent. The classification step had a labeling problem. Two days of prompt work, solved in ten minutes of trace inspection.

Consider a RAG assistant serving a legal research use case. Users start reporting vague, inconsistent answers. The team suspects model temperature drift. A senior engineer looks at cost and token tracking in Phoenix. The trace view shows a pattern: retrieved context had roughly doubled in average token count over the past two weeks. The model wasn't drifting. Its input window was filling up with marginal context that crowded out the relevant passages -- context window saturation.

Consider an enterprise AI platform where two clients ask the same question and receive different answers. A trace comparison tells a different story: a misconfigured tenant-routing rule in the orchestration layer was treating their sessions as equivalent. Not nondeterminism -- a routing bug, invisible to every other diagnostic tool.

The Core Idea

When something breaks in production, the instinct is to change the prompt. The discipline is to inspect the trace first.

A trace is not a metric. Metrics aggregate -- they tell you that error rate is 4% or that latency spiked 200ms. Metrics are useful for knowing that something is wrong. They cannot tell you which step in a multi-component execution chain caused it. That is what traces do.

A trace is the complete, ordered record of every operation your system executed to produce one response: each model call, each retrieval query, each tool invocation, each routing decision. Each individual operation within the trace is called a span.

Where This Hits in Production

The session view is not the same as the trace view -- both are necessary. Phoenix makes a useful distinction between a trace (one request) and a session (multiple related traces stitched together into a conversation thread). Start with the session to find the turn, then drill to the trace for the span.

LangSmith's filtering capabilities are more powerful than most teams use. The filter model supports child-run properties -- you can query for all traces whose tree contains a failed tool call, a specific prompt template version, or a span that exceeded a latency threshold.

Datadog's distinction between querying traces and querying spans matters for multi-component systems. Querying traces gives you end-to-end request views. Querying spans lets you find any nested operation across your entire trace store.

Connecting the Dots

Trace debugging is where the eval stack from earlier topics becomes actionable. RAG triad metrics tell you that context precision dropped. The trace tells you which retrieval span, on which query type, with which document corpus, produced the bad chunks. The pattern holds across topics: metrics locate the anomaly in aggregate; traces localize the cause in the specific execution path.

The trace also reveals which gulf caused the failure -- wrong retrieval results point to Comprehension, missing constraints in the system prompt point to Specification, and consistent failures despite perfect context point to Generalization. The first diagnostic question when reading any trace should be: which gulf am I looking at? A retrieval span returning irrelevant documents is a Comprehension problem you fix in the indexing pipeline. A generation span ignoring a constraint that was never in the prompt is a Specification problem you fix in the harness. A generation span that fails despite perfect context is a Generalization problem that needs a different model, a decomposition strategy, or a fallback. (See Topic 6 for the full framework.)

Teams that instrument traces from day one accumulate a vocabulary of failure modes that makes debugging progressively faster over time. The tenth time you see a context bloat trace, it takes thirty seconds to diagnose. The first time, it takes a day.

In Practice

Start by instrumenting the right data. At minimum, every trace should carry: session ID, user or tenant ID, prompt template name and version, model name and version. Without these, your trace store is an expensive log file.

Build your debugging workflow around the failure category. For logic bugs: look at routing and classification spans first. For retrieval bugs: look at context size and chunk quality. For memory bugs: look at context construction spans. For execution bugs: look at tool call spans for error codes and timeouts.

These four categories map directly to the Three Gulfs framework from Topic 6. When you open a trace, the first diagnostic question should be "which gulf caused this failure?" Retrieval bugs -- wrong documents pulled, irrelevant context dominating the prompt -- are Comprehension Gulf failures: the system misunderstood what information the query actually needed. Logic bugs where constraints were missed or rules weren't enforced point to the Specification Gulf: the requirements existed but never made it into the prompt clearly. And when the system handles your test cases but breaks on novel production inputs, that's the Generalization Gulf showing up in your spans. Traces make gulf diagnosis concrete rather than abstract -- you can point to the exact span where comprehension, specification, or generalization broke down.

Use session views for user-reported failures. A user complaint is almost always a session-level claim. Start with the session to find the turn, then drill to the trace for the span.

Build trace filters that your team reuses. A saved LangSmith filter for "all traces with a child run exceeding 3 seconds" or "all traces tagged with prompt-template=v2 that received negative feedback" is institutional knowledge.