- 01.How to read raw AI outputs and discover failure patterns that no dashboard would surface.
- 02.The two-step method (open coding to axial coding) that turns vague "accuracy problems" into a precise taxonomy of fixable issues.
- 03.Why failure patterns in multi-step agents are fundamentally different from single-turn failures — and how to find the first-bad-step.
The Story
Consider a logistics PM whose AI routing assistant has an accuracy metric sitting at 73%. Leadership wants 85% by end of quarter. Engineering proposes a model upgrade. Data science suggests fine-tuning on historical routing data. Both would take six weeks.
The PM says: "Before we change anything, I want to read 100 traces myself."
She spends four days. No dashboards, no scripts. Just a spreadsheet and the raw traces. For every trace that seems wrong, she writes a brief note. No categories. No analysis. Just what she notices.
By trace 40, she sees patterns. By trace 80, the picture is clear. The failures aren't random. They cluster: weight miscalculation (31%), time zone confusion (24%), multi-stop inefficiency (22%), holiday blindness (12%).
None of these are model problems. Every single one is a data pipeline problem or a specification gap. A model upgrade would have changed nothing. Two data pipeline fixes took two sprints and moved accuracy from 73% to 87%. Zero model changes.
The Core Idea
When an AI product isn't performing, the natural instinct is to look at the dashboard. Accuracy is at 73%. These numbers tell you something is wrong. They tell you nothing about what is wrong or why.
The highest-ROI work a PM can do in the first weeks of any AI product is also the simplest: read the actual outputs. Not a summary. Not an aggregate metric. The raw traces.
Phase 1: Open coding — read each trace and write brief notes about what you observe. No categories. No hypotheses. Just descriptions. The discipline is to observe without judging. Notice patterns before naming them.
Phase 2: Axial coding — step back after 50-100 traces and group your notes into categories. Which failures share a root cause? Which affect the same pipeline component? The goal is a failure taxonomy: 4-7 primary failure modes that account for 80%+ of quality issues.
100 raw traces become roughly 40 open codes, which collapse into 4 axial clusters that account for 89% of the failures.
Here's what makes this practice more than quality assurance: it's product discovery. When the logistics PM found that 12% of failures involved holiday schedules, that wasn't just a bug. It was a product insight — warehouse managers in regions with many state holidays need a feature that doesn't exist yet.
For agents, failure anatomy changes fundamentally. In agentic systems, you need to distinguish three things: outcome failure (did the task actually succeed?), first-bad-step (where did the run first go off the rails?), and visible-damage step (where did the user feel the damage?).
The first-bad-step and the visible-damage-step are often separated by several spans. A data contract failure at step 3 might not produce visible damage until step 8. Reading only the final output misses the root cause entirely.
Recent production guidance increasingly includes measurement failure — the scorer or test is wrong — as a first-class failure type. This matters because teams often treat every failed eval as proof that the product failed. Start by asking where the failure began:
- Product definition: the team did not clearly define the expected behaviour.
- Test data: the example, reference, or expected answer was wrong.
- Knowledge: the system did not find the information it needed.
- Model response: the system misunderstood, reasoned poorly, or invented a claim.
- Tool use: it chose the wrong tool or supplied the wrong details.
- Environment: a tool, database, service, or test environment behaved differently from what the system expected.
- Permission or policy: the system took an action it was not allowed to take.
- Conversation: it forgot or contradicted something from an earlier turn.
- Measurement: the evaluator marked a good result as bad or a bad result as good.
The ninth category is easy to miss. A failed test does not always mean the product failed. Sometimes the test is wrong. In a trace — the linked record of the steps in one run — find the first step that became wrong given the information and rules available at that moment. Later errors may simply be consequences of that first mistake.
Review failure counts by category. If most failures come from missing information, changing the model may be the wrong investment.
Where This Hits in Production
At scale, failure anatomy becomes a sampling design problem. You can't read every trace. The operating model: capture 100% of cheap metadata on every request. Keep full payloads only selectively — 100% for exceptions, guardrail hits, human escalations; 5-10% for high-risk workflows; 1-2% stratified random sample for everything else.
Privacy constraints change the trace review architecture. In healthcare and financial services, production traces contain PII and PHI. The operational sequence: trace capture, redaction/tokenization, review copy. The trace-review workflow in regulated enterprise is "review sanitized copies," not "browse raw logs in a cloud dashboard."
Connecting the Dots
Error analysis is the step most people skip. It's also the most important. More important than the LLM judge. More important than the observability tool. This is where you actually learn what's broken.
And it's faster than people think. First trace: 30 seconds. Second trace: 45 seconds. Third trace: 25 seconds. In an hour, you can review 100 traces.
This is product work disguised as technical work.
Common Mistake
Doing open coding without reaching theoretical saturation.
A PM reads 10 traces, finds 3 error types, and immediately starts building automated judges for those 3 categories. It feels productive. You found bugs. You're building fixes.
What actually happens: you build a quality system that catches the 3 obvious failure modes and is completely blind to the 4th, 5th, and 6th modes that only surface after 60-80 traces. Those silent failure modes are often the dangerous ones.
Keep reading traces until you stop discovering new failure categories. If trace 85 reveals a new type, you haven't reached saturation.Remember This
1. Read 100 raw traces before building any metric, dashboard, or automated eval. The patterns you find will be different from what you expected — and the most valuable discoveries are product insights, not just bugs.
2. For agents, distinguish the first-bad-step from the visible-damage-step. They're often separated by multiple execution spans. Reading only the final output misses the root cause.
3. Most AI product failures are data pipeline problems or specification gaps, not model problems. The PM who reads traces discovers this before the team spends six weeks on a model upgrade that changes nothing.
In Practice: From Bugs to Categories
Here's what failure anatomy looks like when applied to real traces. Open coding turns scattered individual failures into named structural categories you can fix systematically.
Two data-pipeline fixes moved accuracy from 73% to 87% in two sprints. Zero model changes.
References
1. Demystifying Evals for AI Agents — Anthropic
2. Evaluate Agent Workflows — OpenAI
3. LLM Evals: Everything You Need to Know — Hamel Husain
4. Datadog LLM Observability: Patterns — Datadog
5. LangSmith: Prevent Logging Sensitive Data — LangChain
- What product decision does this eval support?
- What evidence would change that decision?
- What does the result still not prove?