- 01.What a trace actually is — the complete execution record of every step your AI took, and why it turns debugging from guesswork into inspection.
- 02.Why "telemetry" is three different capabilities (metrics, events/logs, traces) that vendors market under one word — and why most teams have one while thinking they have all three.
- 03.How the provider telemetry landscape actually works: what you get from OpenAI vs Anthropic, and what that means for your deployment.
The Story
Two weeks of model-level debugging. A five-minute trace fix. A B2B wealth management platform's AI advisor was recommending aggressive growth portfolios to retirees who had asked for capital preservation. The engineering team swapped models, lowered temperature, rewrote the system prompt. Nothing moved.
Someone opened the trace viewer and expanded the spans. The bug was right there: a stray "be aggressive" string had been merged into a hidden sub-prompt template during an unrelated code review. Invisible in the main system prompt. Visible — instantly — in the assembly span.
Without traces: two weeks aimed at the wrong layer. With traces: one line, one diagnosis, one shipped fix. That is the gap this chapter closes.
The Core Idea
When traditional software fails, it usually tells you. An API returns a 500 error. A database query times out. When AI products fail, they usually don't tell you. The API returns 200 OK. The response is well-formatted JSON. The latency is normal. But the answer is wrong.
A trace is the complete record of what happened when your AI system processed a specific request. Not just the input and output — every step in between. Each step is a span — one individual operation recording what went in, what came out, how long it took, and whether it succeeded.
Engineers and PMs hunt for fundamentally different things in the same trace. Engineers focus on technical execution: broken tool calls, retrieval failures, context window overflows. PMs focus on product value: unmet user expectations, confusing responses, tone mismatches. Same trace. Different question. Both essential.
The Telemetry Confusion
"Telemetry" is one of the most overloaded words in AI products. When a vendor says "we support telemetry," a PM needs to ask three separate questions: Do I get metrics? Do I get events/logs? Do I get actual traces with spans?
OpenAI's landscape: The Agents SDK has the strongest native trace story — tracing enabled by default. For direct API, no native trace UI — you use OpenTelemetry instrumentation. Important: tracing is unavailable under Zero Data Retention.
Anthropic's landscape: Claude Code has the richest telemetry surface, exporting to your OpenTelemetry collector. The Agent SDK provides its own telemetry configuration. Direct API relies on OTel instrumentation around the SDK — you bring your own backend.
A trace records every step
your AI took to answer.
API returned 200 OK. System looks healthy.
API returned 200 OK. The answer was wrong.
It records every altitude change, engine reading, and course correction.
Where This Hits in Production
Sampling at scale. At 10,000 queries per day, you can store every trace. At 10 million, full-fidelity storage costs more than the LLM inference itself. Enterprise teams implement tail-based sampling: capture 100% of traces where errors occur, guardrails trigger, or quality flags fire.
Privacy changes the architecture fundamentally. Traces contain the user's actual inputs. In healthcare and financial services, that means PII and PHI. The choice between managed SaaS and self-hosted observability is often made by the compliance team, not the engineering team.
OTel prevents vendor lock-in. By standardizing on OpenTelemetry, teams can swap their backend dashboard without rewriting instrumentation code.
Common Mistake
Instrumenting the model call and calling that "observability."
A team wraps their LLM API calls with telemetry. They can see every prompt sent and every response received. Leadership sees the charts and says "we're covered."
What they've instrumented is one span out of a six-step workflow. The retrieval step, the re-ranking step, the context assembly, the guardrail check, the template rendering — all invisible.
Traces without business metadata and without evaluators are expensive logging. A trace you can inspect one-by-one but can't query by tenant, prompt version, or quality score is visible but not analyzable.In Practice: Reading a Production Trace
12ms
340ms
8ms
1,800ms
45ms
Stale doc ranked higher by vector similarity. Fix: recency weight.
Remember This
1. "Telemetry" is three different capabilities — metrics, events/logs, and traces — sold under one word. Most teams have metrics and think they have observability. Ask which one you actually have.
2. OpenAI ships the agent trace dashboard. Anthropic ships OTel exports and expects you to own the backend. Neither shows you the model's internal reasoning — only the system's execution path. Plan your stack accordingly.
3. A trace without business metadata and without an evaluator attached is expensive logging. If you cannot query "all bad responses for Tenant B on prompt v3 this week," you have records — not observability.
References
1. A Postmortem of Three Recent Issues — Anthropic Engineering
2. AI PM Observability — Aakash Gupta
3. Scribe Cuts Debugging Time with Honeycomb — Honeycomb
4. OpenAI Agents: Tracing — OpenAI
5. Claude Code: Monitoring and Observability — Anthropic