The Limits of Evaluation

The Story

A machine-learning team at an enterprise SaaS company has been building their eval system for nine months. They have dimensional rubrics across accuracy, tone, safety, and completeness. They have a golden dataset of 2,000 curated examples, annotated by domain experts. They have automated LLM judges calibrated against human ratings. They have a CI/CD gate that blocks deployments when scores drop. By every measure this series has taught, they're doing it right. Their pass rate is 96%.

The VP of Product presents this number to the board. "Our AI quality system is mature. We're at 96% across all dimensions." The board is satisfied. The team is proud.

Then three things happen in the same quarter.

First, a new customer segment -- mid-market logistics companies -- starts using the product heavily. Their queries are structurally different from the enterprise finance customers who dominated the golden dataset. Accuracy drops for this segment, but the aggregate pass rate barely moves because the new segment is still small relative to overall volume. The eval suite says 95%. The logistics customers are churning.

This isn't a hypothetical gap. OpenAI's evaluation best-practices guide insists that teams should continuously evaluate, include production and historical data, and grow the eval set over time. The guide exists because the alternative -- a fixed test set that reflects last quarter's users -- is a known failure pattern. A team that curated an excellent golden dataset nine months ago has an excellent snapshot of nine months ago.

Second, the team upgrades their model provider's API version. Internal benchmarks show comparable or slightly better scores. But the team's automated judge -- a strong LLM used for tone and helpfulness evaluation -- also comes from the same provider family. The 2025 scoring-bias research identifies novel biases including rubric order bias and score ID bias in LLM judges. The team's judge may be producing systematically favorable scores for outputs generated by its own model family -- not because of explicit collusion, but because of shared training distributions and stylistic preferences. The eval system isn't lying. It's looking through a lens that has its own distortions, and nobody is evaluating the lens.

Third, the team's competitor publishes a case study claiming their model outperforms the team's model on an agentic coding benchmark. The team tries to replicate the result and gets different numbers. They spend two weeks debugging before discovering that the difference comes from infrastructure configuration -- timeout settings, retry policies, compute allocation. Anthropic's infrastructure-noise study found that configuration alone produced a 6 percentage point gap on Terminal-Bench 2.0 -- more than the gap between top models on the leaderboard. Two agents with different resource budgets and time limits are, in a meaningful sense, not taking the same test.

Three failures, one healthy eval system, one number -- 96% -- that told the board everything was fine. The dataset was stale for a new segment. The judge had its own biases. The infrastructure was part of the score. None of these failures meant the eval system was bad. They meant it was a measurement system, and all measurement systems have limits.

The Core Idea

Every topic in this series has argued that evals make AI products better. Topic 21 showed how to measure more precisely. Topic 22 showed how to learn from failures systematically. Topic 24 showed how to build a compounding improvement cycle. Topic 29 argued that the eval system becomes the executable product spec. All of that stands. Topic 30 doesn't retract any of it.

What Topic 30 adds is the epistemological discipline that makes the rest of the series honest: evals are the best scalable measurement system we have for AI products, but they are still measurements, not reality itself.

They are proxies. Built from chosen tasks, chosen graders, chosen environments, chosen datasets, and chosen definitions of success. Their power comes from making "better" concrete and measurable. Their danger comes from the same source -- the concreteness that makes them actionable also makes them selective. Every eval suite is an editorial decision about what matters, and every editorial decision excludes something.

The teams that get the most value from evals are the ones that hold two thoughts simultaneously: these numbers are the best guide we have, and these numbers are not the whole truth. That's not cynicism. It's the same intellectual posture that separates a good scientist from a technician -- the scientist trusts the instrument and knows its calibration limits.

Going Deeper

The Specification Limit means that whatever you didn't think to measure doesn't appear in the score. Teams operationalize what they can agree on, which systematically excludes what's most contested -- and often most important. OpenAI's agent-skills guide describes an eval as a prompt, a captured run, a small set of checks, and a score. That's good engineering. It's also a confession: the eval measures a selected slice of reality.

The Benchmark-Decay Limit operates at two levels. At the industry level, public benchmarks that your vendor claims to dominate may be partially contaminated or saturated. At the product level, your own golden dataset -- curated last quarter from last quarter's users -- may no longer represent the queries your system actually receives. The 2025 EMNLP survey makes the deeper point: even after the field moved to dynamic evaluation, there are still no standardized criteria for evaluating the dynamic benchmarks themselves.

The Judge Limit is recursive and uncomfortable. The 2025 scoring-bias paper identifies biases most teams never test for: rubric order bias, score ID bias, and reference answer score bias. The position-bias study demonstrates across 150,000+ instances that which candidate appears first affects preference judgments systematically. CALM catalogs at least 12 distinct bias dimensions. Once you automate the judge, you need to evaluate the judge. And the meta-evaluation also has limits.

The Environment Limit means some leaderboard comparisons are partly comparisons of resource allocation rather than model capability. A model given 10 minutes and 3 retries scores differently than the same model given 5 minutes and 1 retry. Both scores are "real" in that they reflect actual performance under those conditions. But comparing them as if they measured the same thing is a measurement error most consumers don't know to look for.

The Observability Limit is the deepest. Anthropic's research on measuring agent autonomy states directly: there is no agreed-upon definition of an agent, providers often cannot reconstruct full sessions from API traffic, and pre-deployment evaluations cannot fully characterize how autonomy emerges in real-world deployment. The eval system -- no matter how comprehensive -- is still looking through a structured lens at an unstructured reality.

Holding Both Truths

The five limits don't add up to "evals are useless." They add up to "evals are instruments, and instruments have calibration boundaries." The practical consequence isn't to distrust your eval system. It's to distrust any single score, any single judge, any single dataset, and any moment where a number replaces an argument.

The best teams hold both truths: this eval system is the most rigorous quality signal we have, and it's a proxy built from editorial decisions we made six months ago with judges we haven't fully audited in an environment that may not match production. Both sentences are true. Competence means acting on the first without forgetting the second.

Where This Hits in Practice

The team with a high pass rate and a growing blind spot. Your eval suite says 96%. A new customer segment is failing on cases your golden dataset doesn't represent. The aggregate number hides the segment-level problem because the new segment is small. The fix: stratify your eval reporting by segment, use case, and time period. An aggregate pass rate is the average temperature of a hospital -- technically accurate, operationally meaningless.

The team comparing models using a single leaderboard. Your vendor says Model X beats Model Y by 4 points on an agentic benchmark. Before acting on that: check whether both models were tested under the same infrastructure configuration, with the same time limits, the same retry policies, the same compute budgets. If you can't verify harness equivalence, the comparison is noise dressed as signal.

The team that trusts the LLM judge without auditing it. Your automated judge produces consistent, scalable scores. Have you tested rubric order sensitivity? Score-label sensitivity? Position bias across candidates? If not, you don't know whether your judge is measuring quality or measuring its own stylistic preferences. Run judge-calibration checks quarterly. Treat the judge as another component to evaluate, not as an oracle.

The enterprise presenting eval scores to the board. A passing score is evidence that the system met certain conditions under certain test parameters with certain judges. Present eval scores with their conditions: "96% on our curated dataset of finance-domain queries, evaluated by GPT-4o as judge, under standard infrastructure settings." That's honesty. "96% quality" is marketing.

Connecting the Dots

From Topic 21 (Deep Rubrics): Rubrics increase measurement resolution. Higher resolution is not the same as completeness. A 7-dimension rubric captures more than a pass/fail check, but it still captures only the dimensions you chose. The specification limit applies at every level of granularity.

From Topic 24 (The Flywheel): The flywheel makes evals smarter over time by feeding failures back into datasets and scoring criteria. But the flywheel can overfit to its own ontology. If the loop only promotes known failure types, known judges, and known environments, the system gets better at what it already knows while unknown unknowns accumulate silently outside the loop.

From Topic 28 (Red Teaming): Red teaming is the operational proof that the frontier moves. Yesterday's adversarial suite becomes today's baseline. Topic 28's continuous adversarial pipeline is the practical defense against the benchmark-decay limit -- but even the best adversarial pipeline is bounded by the attack vectors the team has imagined.

From Topic 29 (Evals as PRD): If the eval system is the executable product spec, then Topic 30 is the intellectual guardrail that keeps that power honest. The scoring function can operationalize product intent. It cannot fully replace judgment about what's worth intending in the first place. Evals measure what the team has decided to value. Someone still has to decide what to value.

Remember This

This series has made the case that evals are the infrastructure of AI product quality -- the rubrics, the datasets, the judges, the experiments, the flywheel, the economics, the platforms, the safety testing, the product spec itself. All of that stands. Topic 30 doesn't weaken it. Topic 30 completes it.

The eval system is the best map you'll build. It converts fuzzy product ambitions into measurable signals. It catches regressions before users do. It turns failures into learning. It makes "better" a decision you can defend with evidence rather than an argument you win with seniority. No serious AI product team ships without it.

But the map is not the territory. The test set goes stale. The judge has preferences. The infrastructure shapes the score. The real world generates behavior that no structured measurement fully captures. The five limits aren't bugs to fix -- they're the structural condition of measurement itself.

The discipline Topic 30 asks for is simple to state and hard to practice: trust your eval system enough to act on it, and distrust it enough to keep questioning it. Present scores with their conditions. Audit your judges. Refresh your datasets. Stratify your reporting. Treat high pass rates as prompts for harder questions, not as permission to stop asking.

Evals are how you navigate. Judgment is how you know when the map might be wrong.