The Eval Flywheel

The Story

A payments company builds an AI assistant that handles refund inquiries. The team runs a solid eval suite: 200 labeled cases, an LLM judge for tone and accuracy, CI gates on every prompt change. After three months, the suite passes at 97%. The team celebrates. They shift engineering focus elsewhere.

Six weeks later, support tickets spike. Customers complain the assistant handles refund timing questions badly — "when will I see the refund?" gets vague, sometimes contradictory answers. The team investigates. The eval suite doesn't have a single refund-timing case. All 200 examples test whether the assistant correctly identifies refund eligibility and processes the request.

The timing question emerged from a product change that added variable processing windows by payment method. Nobody promoted those production failures into the suite. The 97% pass rate was real — and completely irrelevant to the actual failure.

OpenAI's evaluation flywheel cookbook describes exactly this pattern. Once you fix visible failures, "new, subtler failure modes emerge." The eval suite is not wrong. It's stale. And stale suites are worse than absent ones, because they generate false confidence.

A different team — a legal document summarizer — trusts its LLM judge to score accuracy. The judge uses a rubric with four dimensions and produces scores on every production trace. Dashboards look healthy for months. Then a quarterly human calibration session reveals the judge has been systematically under-penalizing a specific failure: summaries that omit material risk factors from the source documents. The judge scores these as 4/5 because the summary is "mostly accurate." A human lawyer scores them as 2/5 because omitting risk factors in a legal summary is not a minor issue — it's the whole point of the document.

A third team hits the plateau from the other direction. Their customer support agent eval suite reaches 99.2% pass rate. They've been diligent — promoting failures, expanding the suite, running weekly. But the suite is now 800 cases, and most of them test patterns from six months ago. The product has added three new features since then. New failures exist in production, but the eval suite literally doesn't ask about them.

Three teams, three different positions on the learning curve, one shared failure: the eval system stopped taking in new information.

The Core Idea

Every eval system has a half-life. The moment you ship it, the product begins to evolve, user behavior shifts, and the world generates failure patterns the suite has never seen. A static eval suite doesn't just stay flat — it actively degrades in relevance. At launch, it covers 90% of the failure surface. Three months later, maybe 70%. Six months later, 50%. The pass rate keeps climbing because the suite keeps measuring the same things. The product keeps changing because reality doesn't care about your benchmark.

The flywheel is the answer: a closed loop where production failures feed the eval system, the eval system gets harder, and the system under evaluation gets better because it's tested against progressively tougher standards.

Stage 1: Capture. Something goes wrong in production. A user gives negative feedback. A monitoring alert fires. A human reviewer flags a trace in the annotation queue. The failure is now a data point. The question is whether it stays as anecdotal evidence or becomes a structured dataset row.

LangSmith's automation rules handle this mechanically: define a filter, set a sampling rate, and specify an action — add to a dataset, add to an annotation queue, trigger an evaluator. Datadog's annotation queues let teams review traces with full span context, apply structured labels, and transfer well-labeled traces into datasets. The shared pattern: don't wait for someone to manually curate. Build the pipeline from production signals to structured data.

Stage 2: Strengthen. Raw failures become eval cases. But the dataset needs curation — not every negative-feedback trace is a real failure, and not every real failure is a good test case. Human annotation separates noise from signal. And crucially, the automated judge itself needs to be checked against these human labels.

OpenAI's flywheel cookbook is specific: calibrate the judge against SME labels using a train/validation/test split, measure TPR and TNR separately, because a judge that always predicts "pass" has high accuracy but zero value. This is the step most teams skip — they update the dataset but never question the judge.

Stage 3: Gate. The updated dataset and recalibrated judge now run as part of every deployment. New prompt versions, model updates, and harness changes all face a test surface that includes the production failures from Stage 1. The key architectural point: the gate should run the same evaluators as production monitoring, against the same versioned dataset. That symmetry means the system is being measured the same way in testing and in reality.

Stage 4: Redirect. As the automated system absorbs more failure families, human reviewers spend less time catching known regressions and more time investigating novel patterns — the failures the system hasn't seen before. Those novel failures become the raw material for the next rotation. The flywheel doesn't eliminate human review. It changes what humans review.

Where This Hits in Production

Dataset versioning is not optional — it's governance. Once production failures become regression cases, changes to those cases alter release decisions. A dataset isn't a spreadsheet; it's a product artifact that determines what ships. "Which dataset version did the release gate run against?" is a question auditors ask.

The flywheel needs multiple graders, not a single score. A single grader creates a single failure mode — if the grader drifts, everything drifts. Multiple graders with different mechanisms (one checks format compliance deterministically, another checks factual accuracy via LLM judge, a third checks safety via rules) make the system robust to any single grader failing.

Start with 20 curated examples, not 2,000. The flywheel doesn't need a massive dataset to begin turning. It needs a disciplined promotion pipeline. A team that promotes 5 production failures per week into a 30-case dataset for the first month has a stronger eval system than a team that spent that month building a 500-case synthetic dataset disconnected from production reality.

Multi-tenant products need selective promotion. Not every production signal belongs in a global dataset. A healthcare tenant's failure patterns are different from a retail tenant's. Mixing them into one global loop dilutes the signal for both.

The Traps

Trap 1: Loop without judge recalibration. The team diligently promotes failures into datasets, expands the suite, runs CI gates — but never checks whether the automated judge is still aligned with human standards. Over months, the judge drifts. It starts over-penalizing certain patterns or under-penalizing others. The dashboard still looks healthy because the judge's own scores are the dashboard.

The fix: quarterly calibration. Pull 50-100 production traces. Have both the judge and 2-3 human experts score them on the same rubric. Compare. If the judge and humans disagree on more than 20% of cases, the judge needs updating.

Trap 2: Flywheel as overhead, not as product. Small teams or early products that build the full loop prematurely spend more time on eval infrastructure than on the product itself. The flywheel's value scales with production volume and failure diversity. A product with 10 users and 3 failure types doesn't need automation rules, versioned datasets, and multi-grader CI gates. It needs a founder reading traces manually and fixing things.

The transition point: when the team can no longer manually review every failure, or when the same failure family recurs after being "fixed," the manual approach has hit its ceiling. That's when the loop earns its infrastructure cost.

In Practice

Week 1: Build the promotion pipeline. Wire the path from production failures to eval dataset. In LangSmith: create an automation rule that routes negative-feedback traces to an annotation queue. In Datadog: configure an annotation queue that receives traces matching your failure criteria. The goal is not a perfect pipeline. It's a pipeline that exists.

Week 2: Seed the dataset from production reality. Pull the last 30 production failures. For each: label the failure type, write the expected correct output, and add to the eval dataset. Version the dataset. This is now your flywheel's starting inventory — 30 real-world failure cases that your eval system previously didn't know about.

Week 3: Establish judge calibration baseline. Take 50 traces (mix of pass and fail). Have the automated judge score them. Have 2 human reviewers score the same traces on the same rubric. Calculate agreement. Document the baseline. This is the number you'll compare against in quarterly recalibration.

Monthly: Promote and prune. Review the annotation queue. Promote 10-20 confirmed failures into the eval dataset. Remove 5-10 lowest-signal cases. Target: 80-90% pass rate on the evolving suite — hard enough to be informative, not so hard it's all edge cases.

Quarterly: Recalibrate the judge. Pull 50-100 fresh production traces. Score with judge + humans. Compare. Update evaluator prompt if disagreement exceeds 20%. Document what changed and why.

An eval suite that isn't absorbing new production failures is decaying. The pass rate climbs while the relevance drops. The flywheel has four stages: capture failures, strengthen judges and datasets, gate deployments, redirect human attention. Every major platform supports all four. The infrastructure exists. The discipline is what's usually missing.