Judging the Judge

The Story

Consider a legal technology company that built an LLM judge to evaluate AI-generated contract summaries. Their judge prompt asked whether each summary was "accurate, complete, and clearly written." The judge performed well in testing -- it caught fabricated clauses, flagged obvious omissions, and penalized summaries that contradicted the source agreement.

Six weeks into production, an outside counsel review surfaced a pattern the judge had missed entirely. The summarizer was consistently omitting limitation-of-liability clauses and indemnification carve-outs from its summaries. Not mangling them -- just leaving them out. The summaries read well, covered the major commercial terms, and were technically accurate about everything they did include. The judge passed them because "accurate and complete" meant, to the judge, that the included content was correct and that the summary covered the main topics. It had no concept that a missing risk disclosure is itself a critical failure. The team spent three weeks retraining the summarizer model before someone asked: what exactly is the judge checking?

The real problem: the judge had poor recall on omission failures. It could recognize wrong information, but not the absence of critical information. The rubric said "complete," but the judge interpreted completeness as topical coverage rather than risk coverage. A contract summary that covers eight of ten sections but skips the two that contain indemnification language is not 80% complete. It's dangerous. The fix required decomposing the single "completeness" criterion into separate axes -- commercial completeness, risk-disclosure completeness, and obligation completeness. Once the judge evaluated each axis independently, its recall on omitted liability clauses jumped from 0.41 to 0.89.

A second team learned a different lesson. They upgraded their judge model -- cheaper, faster -- and saw a sudden 7-point drop in pass rate. The team assumed the shipped assistant had regressed. Manual review later showed the assistant had barely changed. The judge had become stricter on tone and more lenient on missing policy caveats. The team thought the product got worse. What actually happened: the judge changed, and they had no stable reference set to separate judge drift from product drift.

A third team used an LLM judge to compare code-review comments generated by several candidate models. The judge consistently preferred outputs from the same provider family as the judge itself. Recent research identifies this pattern by name: self-preference bias.

Three different teams. Three different judge failures. The legal-tech judge had poor recall on omission failures. The support judge drifted silently after a model change. The code-review judge carried a bias invisible to everyone except the researchers who looked for it. In every case, the team trusted the judge. In every case, the judge was wrong in a way that mattered.

The Core Idea

An LLM judge is a measurement instrument, not a truth source. Once you accept that framing, the workflow changes completely. It stops being "prompt the evaluator until it feels right" and becomes instrument calibration: define the rubric, collect human labels, estimate judge error, monitor drift, and recalibrate when the rubric, product, or judge model changes.

Your LLM judge works like a bathroom scale. A scale that reads 3 pounds light on every measurement is usable -- you can correct for the bias -- as long as you know it's biased. A scale that drifts randomly is unreliable. And a scale you never check against a known weight is worthless, even if it usually seems about right.

Three metrics separate a trustworthy judge from a dashboard decoration:

True Positive Rate (TPR) -- when there's actually a failure, how often does the judge catch it? A judge with 95% TPR catches 95 out of 100 real failures. This is the metric that protects users.

True Negative Rate (TNR) -- when the output is actually good, how often does the judge correctly approve it? A judge with 60% TNR raises false alarms on 40% of good outputs, which means engineers eventually stop trusting the dashboard.

Cohen's kappa -- a chance-corrected measure of agreement. Raw agreement can be inflated when failures are rare. If only 5% of responses fail, a judge that says "pass" every time achieves 95% agreement. Kappa strips out that illusion.

The critical asymmetry: a judge with 88% overall agreement but fail-case TPR below 70% will miss 30% of real failures. If those failures are safety violations, the dashboard says green while users see danger.

Where This Hits in Production

Judge model changes are instrument changes. When you upgrade the judge model -- for cost, speed, or capability -- you are changing the measuring instrument. The operational rule: new judge model, new calibration. Dual-run old and new judges on a stable calibration set before switching, review disagreement clusters manually, and only cut over when the new judge meets TPR/TNR thresholds.

Multi-tenant products need per-tenant judge audits. The same judge prompt can be well calibrated for Tenant A and badly calibrated for Tenant B if their policies, tone constraints, or allowed actions differ.

Annotator onboarding needs gates, not guidelines. Comprehensive onboarding with gold examples, a pilot phase with an 80% accuracy threshold before production, then production consensus labeling with continuing quality checks. Annotators who can't meet the bar don't label production data.

Remember This

1. An LLM judge is a measurement instrument, not a truth source. Calibrate it the way you'd calibrate a scale -- against known weights, periodically, and after every change.

2. Correlation is not calibration. A judge can agree with humans 88% of the time and still miss a third of the failures that matter. Track TPR and TNR separately on the failure slices you care about.

3. When you change the judge model, you change the instrument. New model, new calibration. Dual-run before cutting over, or you'll blame the product for the judge's drift.