Human Review

The Story

Consider a financial services AI assistant with a multi-layer eval stack: automated safety checks, groundedness scoring, topic-relevance evaluation, production monitoring, and CI/CD gates. The team routed any trace with a quality score below 0.7 or a negative user rating to a human annotation queue. The queue launched with 40 items per week. Within two months, 400 items per week were arriving, mostly routine cases flagged for minor relevance dips. Reviewers started marking everything "acceptable" to clear the backlog. The queue was full. The signal was gone.

The calibration problem here is not a technical one. It is the coverage-risk tradeoff in selective prediction: the practice of deciding when a system should defer to human judgment rather than producing an automatic answer. A lower routing threshold catches more bad cases. It also floods reviewers with easy cases until they stop looking carefully at any of them.

Now consider a B2B sales assistant where Legal, Sales, and Engineering each had a representative reviewing flagged outputs. Answers about contract terms for non-standard deals kept returning different review verdicts. Legal flagged them as potentially creating implied commitments. Sales approved them as commercially standard language. Engineering noted the judge had scored them as correct against the rubric. The three teams weren't making different mistakes. They were applying different definitions of "good."

This is the disagreement problem that evaluation research takes seriously. Cohen's kappa -- a statistical measure of inter-annotator agreement that accounts for chance -- is the standard tool for detecting how much disagreement is structural rather than random. When kappa is low, the rubric probably needs revision, not the reviewers.

Third case: a customer support AI whose automated judge consistently scored responses as acceptable. Reviewers in the annotation queue kept leaving freeform notes: "technically correct, but the escalation summary omits the most important caveat" or "factually fine, tone is going to generate a callback." None of this matched any existing eval dimension. The judge wasn't wrong about what it measured -- it just wasn't measuring the thing reviewers were seeing.

Three teams, three different failure modes: a queue that stopped surfacing signal, a rubric dispute that nobody owned, and a failure pattern that only human reviewers could see. All three are part of why human review is a precision system, not an afterthought.

The Core Idea

The full eval stack -- rubrics, judges, RAG triad metrics, conversation evals, agent trajectory scoring, CI gates, production monitoring -- does a lot. It narrows the failure surface, catches known regressions, and scales human-level quality checks to production volume. It does not eliminate ambiguity, resolve contested definitions, or discover failure modes that nobody has yet described in a rubric.

Human review exists for those three things. Not as a fallback when automation fails. As the deliberate layer for cases where automation was never designed to be sufficient.

There are two different modes of human involvement, and confusing them creates operational problems.

Human-in-the-loop means a person approves or rejects a consequential action before it happens. The system pauses, routes the action to a reviewer, and only proceeds once the reviewer decides. This is appropriate for irreversible actions -- sending a communication, processing a financial transaction, modifying account state -- where the cost of an automated mistake is higher than the latency of a human pause.

Human-on-the-loop means a person reviews a sample of outputs after the fact, building feedback that improves the system over time. This is the annotation queue model: LangSmith automation rules routing traces to queues, Datadog annotation queues with full trace context, Scale's annotation workflows. The system doesn't pause; a reviewer examines a curated slice of production behavior and provides structured feedback.

The Feedback Flywheel

The feedback flywheel is the most important thing human review does -- and the most commonly underbuilt.

Reviewers who annotate failing outputs are not just providing quality control. They're creating ground truth. When a reviewer labels a response as "tone is legally risky" or "correct facts, misleading framing" or "right answer, wrong escalation path," they are describing a failure mode that the automated judge hasn't seen -- and potentially doesn't have a dimension for.

That description, systematically captured as a structured reason code, becomes a dataset row. That dataset row becomes a new test case in the CI eval suite. The CI gate now catches that failure class on every future prompt change. The coverage of the automated system expands because of what humans teach it in production.

Where This Hits in Production

Reviewer interface design determines review quality. A reviewer who sees a final response text in a flat list will miss most of the things worth annotating. A reviewer who sees the full trace -- the query, the retrieved context, the tool calls, the model's reasoning steps, the final response, and the prior automated evaluation results -- can diagnose at the level of the span rather than just the output.

Separately: wherever possible, render the artifact in its native form. A chat response belongs in a chat UI. A document summary belongs beside the source document. An email draft belongs in an email layout. Forcing reviewers to interpret formatted text in a generic annotation grid is a design failure that reduces annotation quality.

Cohen's kappa is the first check when review quality feels unreliable. If two reviewers independently labeling a set of outputs produce a kappa below 0.6, at least one of three things is true: the rubric is underspecified, the reviewers need calibration, or the task is genuinely subjective. Most of the time, disagreement reveals that the definition of "good" was never made explicit enough.

The adjudication model is a governance design, not a tooling problem. In enterprise settings where Legal, Sales, Engineering, and domain experts don't share a single definition of "good" -- which is almost everywhere -- the annotation queue doesn't resolve the conflict. It surfaces it. The team still needs to decide who owns the final definition of quality for each failure category. That decision belongs in the governance design before the queue launches.

Connecting the Dots

Topic 20 is the point where the entire stack from Topics 11 through 19 becomes a system rather than a collection of practices. Machine rubrics and judge calibration established that automated evaluation requires human feedback to stay calibrated. Human review is where that calibration actually happens.

RAG triad metrics, conversation evals, and agent trajectory scoring all produce scored outputs. Human review handles the cases where those scores are correct but insufficient -- where the rubric measured the right things but didn't capture the nuance that matters for a specific case. This is where the Completion Fallacy -- the first face of fake success from Topic 2 -- lands in practice: the system marked the task done, the automated score looks fine, but a human reviewer sees that "done" and "done well" are not the same thing.

Production monitoring tells you that quality drifted on some slice of live traffic. Human review decides whether that drift represents a real failure, a false positive from a miscalibrated judge, a rubric gap, or a genuinely new failure mode. Without the human layer, every production alert either gets auto-resolved or sits unexamined. Neither produces learning.

In Practice

Start with the routing design, not the tooling. Before configuring any annotation queue, answer four questions: Which failure types require human-in-the-loop? Which belong in the annotation queue? What routing rules cover each type? Who owns the rubric for each failure category, and who adjudicates disagreements?

Measure inter-annotator agreement before the queue scales. Run a calibration batch: 50 items, two reviewers, same rubric. Calculate Cohen's kappa. If it's below 0.6, revise the rubric and run calibration again before routing production traffic. This is not expensive. It is the difference between an annotation queue that produces signal and one that produces noise at scale.

Design reason codes before you launch. Thumbs up/down tells you that something was wrong. Reason codes tell you what was wrong. Pick the 6 to 8 codes that matter most for your product, train reviewers on them, and make them the primary annotation artifact. That specificity is what turns annotation into dataset enrichment.

Build the feedback loop explicitly. After each annotation batch: identify new failure patterns without a corresponding CI test case, promote confirmed failures into the eval dataset, check whether the automated judge would have caught them, and run the CI suite against the new dataset version to confirm the gate catches the failure. Without this loop, human review is quality control. With it, human review is the engine of continuous improvement.