Failure Anatomy

The Story

Consider a logistics PM whose AI routing assistant has an accuracy metric sitting at 73%. Leadership wants 85% by end of quarter. Engineering proposes a model upgrade. Data science suggests fine-tuning on historical routing data. Both would take six weeks.

The PM says: "Before we change anything, I want to read 100 traces myself."

She spends four days. No dashboards, no scripts. Just a spreadsheet and the raw traces. For every trace that seems wrong, she writes a brief note. No categories. No analysis. Just what she notices.

By trace 40, she sees patterns. By trace 80, the picture is clear. The failures aren't random. They cluster: weight miscalculation (31%), time zone confusion (24%), multi-stop inefficiency (22%), holiday blindness (12%).

None of these are model problems. Every single one is a data pipeline problem or a specification gap. A model upgrade would have changed nothing. Two data pipeline fixes took two sprints and moved accuracy from 73% to 87%. Zero model changes.

The Core Idea

When an AI product isn't performing, the natural instinct is to look at the dashboard. Accuracy is at 73%. These numbers tell you something is wrong. They tell you nothing about what is wrong or why.

The highest-ROI work a PM can do in the first weeks of any AI product is also the simplest: read the actual outputs. Not a summary. Not an aggregate metric. The raw traces.

Phase 1: Open coding — read each trace and write brief notes about what you observe. No categories. No hypotheses. Just descriptions. The discipline is to observe without judging. Notice patterns before naming them.

Phase 2: Axial coding — step back after 50-100 traces and group your notes into categories. Which failures share a root cause? Which affect the same pipeline component? The goal is a failure taxonomy: 4-7 primary failure modes that account for 80%+ of quality issues.

Here's what makes this practice more than quality assurance: it's product discovery. When the logistics PM found that 12% of failures involved holiday schedules, that wasn't just a bug. It was a product insight — warehouse managers in regions with many state holidays need a feature that doesn't exist yet.

For agents, failure anatomy changes fundamentally. In agentic systems, you need to distinguish three things: outcome failure (did the task actually succeed?), first-bad-step (where did the run first go off the rails?), and visible-damage step (where did the user feel the damage?).

The first-bad-step and the visible-damage-step are often separated by several spans. A data contract failure at step 3 might not produce visible damage until step 8. Reading only the final output misses the root cause entirely.

Where This Hits in Production

At scale, failure anatomy becomes a sampling design problem. You can't read every trace. The operating model: capture 100% of cheap metadata on every request. Keep full payloads only selectively — 100% for exceptions, guardrail hits, human escalations; 5-10% for high-risk workflows; 1-2% stratified random sample for everything else.

Privacy constraints change the trace review architecture. In healthcare and financial services, production traces contain PII and PHI. The operational sequence: trace capture, redaction/tokenization, review copy. The trace-review workflow in regulated enterprise is "review sanitized copies," not "browse raw logs in a cloud dashboard."

Connecting the Dots

Error analysis is the step most people skip. It's also the most important. More important than the LLM judge. More important than the observability tool. This is where you actually learn what's broken.

And it's faster than people think. First trace: 30 seconds. Second trace: 45 seconds. Third trace: 25 seconds. In an hour, you can review 100 traces.

This is product work disguised as technical work.

Remember This

1. Read 100 raw traces before building any metric, dashboard, or automated eval. The patterns you find will be different from what you expected — and the most valuable discoveries are product insights, not just bugs.

2. For agents, distinguish the first-bad-step from the visible-damage-step. They're often separated by multiple execution spans. Reading only the final output misses the root cause.

3. Most AI product failures are data pipeline problems or specification gaps, not model problems. The PM who reads traces discovers this before the team spends six weeks on a model upgrade that changes nothing.

In Practice: From Bugs to Categories

Here's what failure anatomy looks like when applied to real traces. Open coding turns scattered individual failures into named structural categories you can fix systematically.