Production-Grade Trace Scoring

Three Teams. Three Dashboards. Three Quiet Failures.

Story one. The sales team that built beautiful briefs nobody used. A B2B sales team shipped an AI that wrote a one-page customer brief before every call. Their eval team graded each brief on four rubrics — faithful (no invented facts), coherent (well organized), honest (no over-claiming), and helpful (covered every standard section: funding, leadership, recent press, tech stack). After two months, every rubric averaged above 0.90. The dashboard was green. They rolled it out to twenty reps.

Four weeks later, the reps had quietly stopped opening the briefs. The post-mortem was unanimous: the briefs were technically excellent and operationally useless. They never surfaced the one thing a rep needs before a discovery call — the specific buying signal the customer had dropped in the last ninety days, and a sharp opening question that would surface budget. The helpful rubric only asked "did the brief cover the standard sections?" It never asked the question the rep was paying for: "did this brief help me run a better call?"

Moral: a rubric measures what it was written to measure. If nobody wrote down the user's real goal, no rubric will catch its absence.

Story two. The doctors' notes that passed every quality check and triggered an insurance audit. A clinical-scribe team — AI that listens to a doctor-patient visit and writes the medical note — was a step ahead of most. Before trusting their AI to grade itself, they ran a check on the grader. They took 300 real notes, had clinicians score each one by hand, then had the AI judge score the same 300, and confirmed the two agreed about 9 times out of 10. In eval language, the judge was "calibrated." The dashboard stayed green for six months.

In month seven, a doctor raised a flag. The insurance company that pays the hospital was running its own audit. Insurance audits don't ask "is the note medically correct?" — they ask "does this note prove the visit actually happened the way it's billed, in the patient's own situation, in the doctor's own words?" The AI had quietly settled into a few comfortable sentence patterns. To a clinician, the notes still read fine. To the insurer's audit team, hundreds of notes suddenly looked copy-pasted — a red flag for fraud. The hospital spent 200 hours rewriting notes and barely avoided a formal finding.

The mistake was who wrote the scoring rubric. Clinicians wrote it, so it asked clinical questions: is the diagnosis right, are the medications listed, is the history accurate? The people who actually deal with insurance audits — the revenue-cycle team — were never in the room. So "does this note look templated to a payer?" was never a question the AI judge could ask, and never a number on the dashboard.

Moral: the people who write the rubric decide what your dashboard can see. Whoever isn't in the room becomes a blind spot.

Story three. The legal AI that aced every metric and still lost every head-to-head pilot. A legal-research startup did everything the textbook tells you to do. They checked their AI judges against human lawyers to make sure the judges agreed with experts. They broke their scores down by case type, jurisdiction, and matter size, so a weakness in, say, small employment cases couldn't hide inside a strong average. They threw deliberately tricky and misleading questions at the system to see where it broke. They swapped one fact in a question and checked whether the answer changed the way a lawyer would expect. Before every prompt change, they wrote down the improvement they expected, then measured whether they actually got it. By any technical standard, the evaluation program was excellent.

Then law firms started running side-by-side trials: same questions, their tool versus a smaller, less-funded competitor, partners picking the winner. They lost every trial. The competitor scored lower on every metric the startup published — less complete, less precise, less polished.

The interviews explained it. A partner doesn't ask a research question to receive a beautifully written answer. They ask it to move a client matter one step forward — toward a filing, a call, a decision. The competitor's answers ended with a line like "the next thing to ask your client is whether the contract was signed before or after the merger closed." The startup's answers ended with a perfect summary of the law. Technically better. Practically useless. Nothing in their evaluation stack measured "did this answer move the matter forward?" — because no one had ever made forward motion a thing the AI was graded on.

Moral: measuring the wrong thing very precisely is still measuring the wrong thing.

Three teams. Three levels of sophistication. One shared failure. Every trace was being scored on the answer itself — graded against vague quality axes like faithful, coherent, helpful — but none of those axes ever measured whether the user's actual goal got done. Every layer of rigor on top of the wrong foundation just polishes the wrong work harder.

That is the topic of this chapter. A production-grade eval stack starts with the right unit of scoring, then layers the seven disciplines on top. Start with the wrong unit and the disciplines move you faster in the wrong direction.

Two Ways to Score the Same Trace

There are two ways to score an AI output. Most teams are running the worse one and don't know they had a choice.

Way A — Quality-Modifier Scoring. Score the answer on a handful of axes — faithful, coherent, honest, complete — and report the average. This is what every major eval framework ships by default. The axes are real signal. But they are modifiers, not goals.

"But my rubric has a helpfulness axis — doesn't that catch it?"

No, and the reason is structural. Every "helpfulness" judge in every mainstream eval framework — RAGAS, OpenAI Evals, Anthropic's HHH, the LangSmith defaults — works the same way: it looks at the prompt and the response, and asks an LLM judge "is this response helpful for that prompt?" Four things follow from that setup, and all four are problems:

1. The judge only sees text, not outcome. It never sees whether the rep closed the call, whether the partner filed the brief, whether the note survived the audit. Helpfulness is judged on how the answer reads, not on what it caused.
2. It assumes the prompt equals the goal. A partner types "summarize the precedent on X" — but their actual goal is "move this matter one step forward." The judge grades against the typed question, never against the unstated job. A perfect answer to the literal prompt scores 1.0 while the user's real goal stays untouched.
3. LLM judges optimize for surface signal. Given only prompt + response, the cheapest features for a judge to lock onto are relevance, completeness, fluency, and tone. Goal completion is invisible at that layer, so the judge silently substitutes "well-written and on-topic" for "helpful."
4. Goal completion is a session-level signal, not a trace-level one. It usually needs the next user action, a downstream tool call, or a real-world outcome (a booking, a filing, a payment). A single-trace "helpfulness" score has no access to any of those, so by construction it cannot contain the answer.

That is why adding helpful as a fifth axis doesn't save Way A. It is still a property of the text, scored by a judge that only sees the text. You cannot bolt goal-completion onto a rubric that was designed to grade prose. Way B isn't a new axis — it's a different unit of measurement.

Way B — Goal-First Scoring. Score whether the user's actual goal was achieved as the primary thing. Everything else — faithful, voice, latency, cost — is a modifier that only matters if the goal was even partly achieved. If goal completion is zero, the trace score is zero, no matter how nicely the answer was written.

The difference is structural. Way A is adding things up: a high score on one axis rescues a low score on another. Way B is gating: nothing rescues a failed goal. You cannot polish your way past a useless answer.

Think of it like a Yelp review that starts with did the meal arrive? If it didn't, the review caps at one star, no matter how nice the plating photo looks on the chef's Instagram. The chef can earn extra stars for presentation, service, ambiance — but only on top of a meal that actually showed up. Rating presentation alone is how restaurants end up with five-star plating photos and one-star revenue.

The Math, in Plain English

The standard formula in most eval frameworks looks like this:

trace_score  =  w₁·faithful + w₂·coherent + w₃·honest + w₄·helpful

Translation: add up four scores, weight them, ship the average. The problem: if helpful is 0.65 (the answer addressed the wrong question) and the other three are above 0.90, the average comes out to 0.87. The dashboard says green. The user got nothing they needed.

The production-grade formula is structurally different:

trace_score  =  goal_completion  ×  (1 + α · modifier_score)  ×  robustness

Translation, line by line:

• goal_completion — between 0 and 1. Did the user get what they came for? If this is zero, the entire trace scores zero.
• modifier_score — between 0 and 1. The old quality axes (faithful, coherent, honest, fast, cheap) rolled into one number.
• α — a small tuning knob, usually 0.3 to 0.5. Big enough to separate good from great. Too small to let modifiers rescue a failure.
• robustness — between 0 and 1. Downweights traces where the system only got lucky on this exact phrasing.

The key property: if goal_completion is zero, the whole product is zero. No amount of polish on the modifiers can rescue a failed goal. A trace can only earn quality credit for serving the user — and then doing it well.

"But How Do You Know What the User's Goal Was?"

The hardest objection to goal-first scoring is practical. In a benchmark with a labeled answer, the goal is the answer. In a real production trace, the goal is whatever the user was trying to do — which they did not always say, and which the system did not always capture.

There are four sources of goal signal, from easiest to hardest:

1. Ask the user. Add a one-line field: "what does success look like?" The cheapest option. Some teams (Stainless for API docs) let the user pick a goal type — "working code example" vs "conceptual explanation" — and score against each goal differently.
2. Classify the intent. Run a small classifier that maps every input to a goal type — lookup, comparison, recommendation, debugging, drafting, decision support. Each goal type then has its own success criteria. This is the most common practical pattern in 2026.
3. Watch what the user does next. Did they copy the answer? Ask a follow-up that built on it, or one that re-asked the same question in different words? Close the session, or reformulate? Noisy on any one trace, powerful in aggregate. If your judge says goal completion is 0.85 and downstream behavior says users reformulated eighty percent of the time, your judge is wrong about goal completion.
4. Verify the outcome. The strongest signal — only available for agents that act. Did the action change the world the way the user wanted? Was the flight booked, at the right price? Did the contract pass legal review? The world tells you the answer; no judge required.

Mature stacks use at least two of these — a stated or classified goal at the input, and either downstream behavior or outcome verification at the output. Single-source goal scoring is fragile. Combining sources triangulates the goal the same way combining judges (L2-T13) triangulates a quality score.

Seven Disciplines on One Foundation

The seven disciplines the eval field talks about — judge calibration, distribution checks, slice analysis, negative space, adversarial robustness, counterfactual consistency, statistical honesty — are not seven independent metrics. They are six layers of defense above the foundation, each protecting against a specific failure mode.

A team that runs all seven without thinking about how they layer is doing rigor theatre. A team that runs them as a pyramid is doing research-grade evaluation.

The Seven Layers, Each in One Breath

Layer 0 — Goal completion. The foundation. Without it, every layer above accelerates you in the wrong direction. If goal completion is not the gated primary axis, no amount of judge calibration or slice analysis will save the system.

Layer 1 — Judge alignment. Do we trust the judges? Before any score is meaningful, measure how often the AI judge agrees with a human expert on the same trace. (See Judging the Judge (L2-T12).) Most teams calibrate one judge, add a second axis, and assume it inherits the calibration. It doesn't. Each axis needs its own calibration. Any axis below 0.65 agreement is dropped from the production score until it earns its place.

Layer 2 — Distribution check. Does the eval set look like production? The most common silent failure of mature eval programs is eval-set staleness. The team launched with a curated test set, the product evolved, user behavior evolved, the test set did not. A year later the scores are passing while the product is leaking. Move: monthly check on how different production questions look from your test set; auto-refresh the set when it drifts too far.

Layer 3 — Slice analysis. Averages always lie. A 92 percent score can mean uniformly good or 96 percent on lookup, 70 percent on multi-step questions. The fix priority comes from the worst slice, not the average. Move: the dashboard's default view is the worst-performing slice, not the headline number.

Layer 4 — Negative space. Most evals score answers given. They ignore refusals, abstentions, and clarifying questions. Knowing when not to answer is a first-class capability. Three sub-suites: should-refuse (out-of-scope topics), should-clarify (underspecified inputs), should-soften (topics where the source has touched but not taken a position). Failure to refuse blocks the deploy.

Layer 5 — Adversarial robustness. Separate from quality, because the failure modes are categorical, not gradient. Prompt injection either works or it doesn't. Forty to sixty cases across six attack categories. A single passing attack blocks the deploy. (See Red Teams (L3-T28).)

Layer 6 — Counterfactual consistency. A good answer to a question should not change dramatically if you reword the question. Take a sample of test questions, generate three paraphrases of each, run all four, measure two things: do paraphrases produce equivalent answers (consistency), and do meaning-changing twins get correctly different answers (sensitivity)?

Layer 7 — Statistical honesty. A score on fifty questions has wide statistical uncertainty. Reporting "90.0% faithfulness" on fifty traces implies a precision that isn't there. Most teams iterate prompts based on noise — they tell themselves the score went up when the change is inside the confidence interval of the previous version. Every aggregate gets a confidence interval; every prompt change pre-registers the expected delta.

Where This Hits in Production

The cost is real. A goal-first stack with all seven layers needs an intent classifier, per-axis judge calibration sets, monthly drift monitoring, slice infrastructure across at least six dimensions, three negative-space sub-suites, an adversarial suite of forty to sixty cases, a counterfactual suite of sixty paraphrased traces, and confidence intervals on every aggregate. The labeling cost alone is fifteen to twenty thousand dollars in expert time to bootstrap, plus two to three thousand per quarter to maintain. (See Eval Economics (L3-T26).)

Most teams should not build all seven layers from day one. The right path is Layer 0 first — get goal completion as the primary axis, even with one judge and no calibration. Then Layer 1 — calibrate the judge. Then Layer 3 — slice on the dimensions that drive your product's failures. Then Layer 4 if your bot is grounded in a domain. Then 2, 5, 6, 7 as the system matures and the stakes rise.

Multi-tenant systems make this harder. A legal tenant's recommendation goal is not a healthcare tenant's recommendation goal. The intent classifier needs per-tenant variants. The judge prompts need tenant-specific overlays. Without it, the trace score is meaningful for the average tenant and meaningless for any single one — which is the worst of both worlds. The moat is operational: a tenant-aware scoring stack takes months to build and is what separates eval programs that hold up in enterprise procurement from ones that win demos and lose contracts.

Cost-per-goal-achieved is the executive-friendly metric. Once goal-first scoring is in place, the natural CFO question is: what does it cost us to achieve one user goal? That number is (tokens + latency-seconds × cost-per-second + infra) ÷ (sum of goal_completion across all traces). It aligns the eval system with the business model in a way no quality-modifier metric can. No board has ever asked about faithfulness. Every board asks what it costs to deliver value.

One Trace, End to End

An equity analyst types: "Pull me the latest on Datadog — they're reporting tomorrow and I want to know what's changed since last quarter that might surprise." The system runs a multi-step trace: query expansion, retrieval from internal notes and SEC filings, web search, synthesis. It produces a 600-word brief with citations.

Here is what each layer produces on this single trace.

Putting it together:

goal_completion   = 0.75
modifier_score    = 0.5 × 0.93 (faithful) + 0.5 × 0.72 (complete) = 0.825
                    (voice-fidelity excluded — below kappa threshold)
robustness_factor = 0.91
α                 = 0.3

trace_score = 0.75 × (1 + 0.3 × 0.825) × 0.91
            = 0.75 × 1.2475 × 0.91
            = 0.852

The trace scores 0.852 — solid, not elite. The actionable insight is at Layer 0: the missed regulatory risk dropped goal completion from a likely 1.0 to 0.75. The fix is not a judge-prompt change. The fix is adding a regulatory-news retrieval step to the pipeline, tested against the regression set with a pre-registered expected lift of +0.10 on goal completion.

Compare what a standard quality-modifier system would have produced for the same trace: average of (0.93 + 0.72) ÷ 2 = 0.825. The dashboard would have said this was a good trace. The analyst would have walked into the earnings call without knowing about the FTC inquiry. The system would have looked healthy while quietly producing a polished failure on a question that mattered to revenue.

Connecting the Dots

From L2-T11 (Machine Rubrics): rubrics taught you to write criteria judges can follow. But the most consistent judge in the world, following a rubric that measures the wrong unit, is a beautifully calibrated dial pointed at the wrong number.

From L2-T12 (Judging the Judge): you made the judge trustworthy. A trustworthy judge of an irrelevant axis is trust spent on the wrong purchase.

From L3-T21 (Deep Rubrics): you added severity tiers. Severity tiers on a non-goal axis multiply rigor on the wrong thing.

From L2-T16 (Agent Evals): the autonomy spectrum makes goal-first scoring more important, not less. A chatbot can sometimes get away with quality-modifier scoring because a human in the loop catches goal failures the eval missed. An autonomous agent has no human catch. The eval is the only thing between the agent and the user-goal failure.

Remember This

1. The unit of scoring is the user's goal, not the answer's quality. A faithful, coherent, honest answer to the wrong question scores zero in a system that protects users. If your scoring formula does not gate on goal completion, your dashboard is lying about whether your product works.
2. The seven disciplines are a pyramid, not a checklist. Layer 0 is goal completion; layers 1 through 7 stack on top. A team running four layers on top of Layer 0 beats a team running all seven without it.
3. Cost-per-goal-achieved is the executive-friendly metric. It collapses faithfulness, latency, cost, and goal completion into one number that aligns engineering with business value. No board has ever asked about faithfulness; every board asks what it costs to deliver value.

If goal completion is zero, the trace scores zero — no matter how the modifiers did. A trace cannot earn quality credit for failing the user. It can only earn quality credit for serving the user — and then doing it well.