Evaluating the Harness

The Story

A wealth management platform uses AI to generate client portfolio summaries for financial advisors. The summaries combine current holdings, market performance data, regulatory updates, and historical recommendations. Quality was strong at launch — advisors rated 89% of summaries as "ready to send" with minimal editing.

Six months later, "ready to send" dropped to 71%. Advisors were editing more summaries, flagging more inaccuracies, and occasionally catching summaries that referenced the wrong client's holdings — a compliance nightmare in financial services.

The product team's response: rewrite the prompts. The system prompt was revised three times over four weeks. After each revision, quality on the eval set improved to 88%. Quality in production stayed at 72%.

A new engineer joined and asked a question nobody had asked: "What's in the context window when these bad summaries get generated?" She built a simple trace logger that captured the complete assembled context. Within two days, she found the pattern.

The retrieval system was pulling market data from a stale index. The real-time market feed had changed its API schema three months ago. The data ingestion pipeline was failing silently — no errors, just empty updates. The vector index hadn't received fresh market data in 94 days. Every summary was using market data from January.

Four weeks of prompt rewrites. Zero improvement in production. Because the team was treating a harness problem as a model problem — and their evaluations couldn't tell the difference.

The Core Idea

Most AI product teams evaluate one thing: was the model's output good? They compare the response to a reference answer, have human raters score helpfulness, use LLM-as-judge to automate quality scoring. All of these evaluate the final output. None of them evaluate whether the harness assembled the right context.

The distinction is critical:

Model evals test: "Given what the model saw, was the response good?"

Harness evals test: "Did the model see the right things?"

Both are necessary. Neither alone is sufficient. A model can produce a perfect response to wrong context — accurate reasoning from incorrect premises. A harness can assemble perfect context and the model can still produce a mediocre response.

Stage 1: Retrieval evaluation. Did the pipeline return the right documents? Precision (of documents retrieved, how many were relevant?) and recall (of all relevant documents, how many were retrieved?). The PM defines the acceptable balance based on the product.

Stage 2: Assembly completeness. Was a constitution loaded? The correct version? Were knowledge chunks included within the token budget? Was memory loaded for the right user?

Stage 3: Tool selection accuracy. Were the right tools loaded for the detected intent? Misrouted tools waste tokens and can cause wrong actions.

Stage 4: Template compliance. Did the output conform to the expected schema? Non-conforming responses often indicate a context assembly issue, not a model issue.

Integration with CI/CD pipelines. Harness evaluations aren't just for production monitoring — they're deployment gates. Before any change to the harness reaches production, the CI/CD pipeline should run the harness eval suite and fail the deployment if metrics drop below thresholds.

This prevents the scenarios that haunt production teams: a knowledge base configuration change that silently breaks retrieval. A constitution update that exceeds the token budget. A tool definition change that causes misselection. These regressions are caught before they reach users.

The feedback loop. Observability captures production traces. Quality reviews identify poor outputs. For poor-quality outputs, the trace reveals whether the harness assembled the right context. Cases where the harness assembled wrong context become new eval examples. The expanded eval suite prevents the same failure class from recurring.

Where This Hits in Production

The prompt-vs-harness diagnostic. When quality drops, the first diagnostic question is: model problem or harness problem? Without harness evals, teams default to "rewrite the prompt." With harness evals, the diagnostic is specific: retrieval precision dropped from 87% to 64% because the index hasn't been updated in three weeks.

Retrieval regression detection. Knowledge bases change constantly. New documents are added. Embedding models are upgraded. Any of these changes can silently degrade retrieval quality. A weekly retrieval eval catches regressions before they accumulate into user-visible quality problems.

Multi-tenant harness verification. In B2B, harness configuration varies per tenant. Each tenant has a different constitution, different knowledge base, different tool permissions. Harness evals should run per tenant to verify that configuration changes for one tenant don't break another.

Remember This

1. Model evals test "was the answer good?" Harness evals test "did the system assemble the right context?" Both are necessary. A perfect model with a broken retrieval pipeline produces wrong answers from correct reasoning — and prompt rewrites won't fix it.

2. Per-stage evaluation — retrieval precision/recall, assembly completeness, tool selection accuracy, template compliance — isolates where quality problems originate. This turns "the AI is wrong" into "retrieval precision dropped to 64% because the index is stale."

3. Harness evals belong in CI/CD gates AND in continuous production monitoring. Code-level gates catch deployment regressions. Continuous monitoring catches data-level regressions that happen between deployments.