Deep Rubrics

The Story

A coding assistant ships with a clean eval gate. Output compiles, JSON is valid, classification labels match expected answers. Green across the board. Three weeks later, a senior engineer reviews a sample of the AI's code contributions and flags half of them: brittle error handling, no input validation, zero documentation, security patterns from 2018. The code "worked" by every automated check. No senior engineer would approve it.

"Does it compile" is not the same question as "would a senior practitioner accept this." The gap between those two questions is exactly where deep rubrics live.

A legal AI assistant generates compliance memos for a financial services firm. The team's eval judge scores overall quality on a 1-5 scale. Most outputs land at 4. Then a regulatory audit asks for line-by-line verification. An outside counsel finds missing risk disclosures, inadequate precedent citations, and structural issues that any experienced attorney would catch -- none of which moved the overall score because they were averaged away.

XpertBench contains 1,346 expert tasks across 80 professional categories, each with rubrics of 15-40 weighted checkpoints. Even with that granularity, leading models reach only about 66% peak success, with a mean around 55%. Once the work looks like real professional output, "good overall" is too blunt to be useful.

A product team builds a rich rubric with 12 dimensions, each scored 1-5, and feeds it to their LLM judge. Then they discover that the judge's rubric-level scores disagree with expert human judgments on about 44% of the hard cases. RubricEval tested exactly this: on its Hard subset, even GPT-4o reached only 55.97% accuracy at the rubric level.

Adding more rubric dimensions does not automatically make evaluation more trustworthy. Unless the judging process itself is calibrated against human ground truth, richer rubrics can produce more confident wrong answers.

The Core Idea

Pass/fail evaluation answers one question: ship or don't ship. That's necessary. It's not sufficient. Once an AI system produces work that looks like professional output -- code, legal analysis, medical summaries, financial recommendations -- the quality bar is inherently multi-dimensional.

"Correct" is one dimension. Secure, maintainable, well-structured, properly cited, appropriately caveated, tone-calibrated -- these are separate dimensions, and a failure in any one of them can make "correct" output professionally unacceptable.

Where This Hits in Production

The diagnostic scorecard vs. the release gate -- most teams conflate them. Deep rubrics produce diagnostic signal: per-dimension scores that tell you where quality succeeds and where it fails. Release gates produce binary decisions: ship or don't ship. Braintrust's architecture makes this explicit: scorers produce continuous scores (diagnostic), while per-scorer pass thresholds produce binary verdicts (gating).

Orthogonality matters more than quantity. AdaRubric's central insight: evaluation dimensions should be orthogonal. "Is the response clear?" and "Is the response easy to understand?" are two rubric items that measure one thing. They create an illusion of thoroughness while tripling the weight of a single dimension. If removing a dimension loses no diagnostic information, remove it.

Calibration is the invisible requirement. LLM-Rubric's contribution: calibrate across multiple rubric-question distributions to predict human judgment. In dialogue evaluation, nine rubric questions around naturalness, conciseness, and citation quality predicted overall user satisfaction with RMS error under 0.5. A rubric you don't calibrate against human ground truth is a rubric you're hoping works.

Multi-tenant products need rubric layering. A healthcare tenant cares about clinical accuracy and liability language. A retail tenant cares about product accuracy and return policy compliance. Ragas' split between domain-specific and instance-specific rubrics acknowledges this directly. The pattern: common score semantics, tenant-specific evaluation behavior.

In Practice

Start with the inspection checklist, not the rubric architecture. Before building anything, write down what a senior practitioner would check when reviewing this AI's output. That list is your rubric seed. Formalize it later.

Make each dimension independently scorable. Define 3-5 scoring levels with concrete anchor examples. Write 2-3 test cases per dimension. Verify the judge can score each dimension with kappa > 0.6 against human labels.

Version your rubrics like you version your prompts. When the rubric changes, that's a spec change that affects every experiment, dashboard, and release policy that references it. Treat rubric changes with the same governance rigor as prompt changes.

Calibrate quarterly. Every quarter, pull 50-100 production outputs. Have both the LLM judge and 2-3 human experts score them. Calculate kappa per dimension. Any dimension where kappa drops below 0.6 gets revised, gets more anchor examples, or gets retired.