The Harness-Eval Contract

The Story

Two teams at the same enterprise SaaS company -- call them Team Atlas and Team Beacon -- both ship AI-powered features in the same quarter. Atlas builds a contract analysis tool for their legal product. Beacon builds a customer health scoring system for their success product. Both teams use Claude Sonnet. Both have competent engineering. Both have experienced PMs.

Eighteen months later, Atlas's product quality is consistently improving. Their accuracy metrics trend upward quarter over quarter. When quality dips, they diagnose and fix the root cause within days. Their PM presents quarterly quality reviews with specific metrics: retrieval precision, assembly completeness, end-to-end quality scores, cost per resolved query. The executive team trusts the numbers because they can trace any metric back to its source data.

Beacon's product quality oscillates. Good months follow bad months with no clear pattern. When quality dips, the team debates whether it's a model issue, a data issue, or a prompt issue. Debugging takes weeks. Their PM reports quality as a single number -- user satisfaction -- that moves in ways nobody can explain. The executive team has started asking whether the feature is viable.

The difference isn't engineering talent or model capability. It's that Atlas built a harness-eval contract -- a formal specification of what the harness exposes for evaluation -- before writing a line of product code. Beacon built the harness and the evaluation system as separate projects, by separate teams, with no shared interface specification.

The Core Idea

The harness-eval contract is a specification document that defines what the harness must expose -- in what format, at what granularity, with what latency -- for the evaluation system to function. It's the interface between the system that builds context and the system that judges quality.

This sounds like a technical infrastructure concern. It's actually a product architecture decision with strategic consequences.

Every AI product has two quality loops:

Loop 1: Engineering health monitoring. Is the system up? Are response times within SLA? Are error rates acceptable? This loop is owned by engineering and monitored through standard observability tools. It answers: "Is the system working?"

Loop 2: Product quality evaluation. Are the responses accurate? Are users satisfied? Is the product getting better over time? This loop is owned by product/quality teams and monitored through eval frameworks. It answers: "Is the product good?"

In most organizations, these loops are completely separate systems, built by different teams, consuming different data, answering different questions. The gap between them -- "the system is healthy AND the product quality is poor" -- is the dead zone where the hardest problems live.

The harness-eval contract bridges this gap by specifying that the harness must emit the data both loops need -- in a shared format, from a single source.

Think of it like the diagnostic port on a modern automobile. The car's computer systems generate detailed telemetry -- engine timing, fuel injection rates, sensor readings, error codes. The mechanic's diagnostic tool plugs into a standardized port (OBD-II) and reads that telemetry to diagnose problems. Without the port, the mechanic listens to the engine and guesses. The OBD-II standard IS the contract between the car's systems and the diagnostic tools. The harness-eval contract is the OBD-II port for your AI product.

What the harness must expose. The contract specifies five categories of data:

1. Structured traces per CONTEXT layer. Every assembly stage produces a tagged span with start/end timestamps, success/failure/degraded status, token count, stage-specific metadata, cache hit/miss status, and fallback triggered indicators.

2. Token counts per component. The total token count of the assembled context, broken down by CONTEXT layer. This enables cost attribution, budget compliance monitoring, and optimization targeting.

3. Latency per stage. Wall-clock time for each assembly stage, including parallel execution groups. This enables performance regression detection and bottleneck identification.

4. Complete assembled context at inference time. The full context window -- everything the model saw -- captured and retrievable for any given trace. When a response is wrong, the evaluator can pull the exact context and determine whether the model made an error or the harness made an error.

5. Tool call results and memory load details. When tools are invoked during generation, the call parameters and results must be traced. These execution-time details capture the dynamic behavior of the system during generation.

Where This Hits in Production

Vendor evaluation. When evaluating harness frameworks (LangChain, LlamaIndex, Haystack, custom), the harness-eval contract becomes the evaluation criterion. Does the framework emit structured traces per assembly stage? Can you access the complete assembled context? A framework that's easy to build with but hard to observe is a liability at production scale.

Compliance documentation. In regulated industries, the harness-eval contract doubles as compliance documentation. Financial regulators want to know: what information did the AI use to make this recommendation? The contract specifies that this information is captured, retained, and retrievable -- meeting audit requirements through the same infrastructure that serves product quality evaluation.

Team alignment. The contract eliminates a common friction between engineering and product teams. Without it, conversations go: "The AI quality is bad." "The system is healthy." "Then why is quality bad?" "We don't know." With the contract, both teams look at the same data. The conversation becomes: "Retrieval precision dropped 12% this week." "The knowledge base sync job failed Tuesday." "That correlates. Fix the sync job and re-run the retrieval eval to confirm."

In Practice

Atlas's harness emits structured traces with tagged spans for every CONTEXT layer. Their eval system consumes those traces directly. When retrieval quality drops, the retrieval span metrics show it immediately, and the eval system flags the specific queries where retrieval failed. The PM sees "retrieval precision dropped 8% this week, concentrated in tenant XYZ's knowledge base" -- not "user satisfaction is down."

Beacon's harness logs requests and responses. Their eval system runs monthly batch evaluations against a golden dataset. When quality drops, the eval system detects it weeks later. The debugging question -- "was this a harness problem or a model problem?" -- takes days to answer because the eval system can't see inside the harness.

Remember This

1. The harness-eval contract bridges the gap between engineering monitoring ("is the system healthy?") and product evaluation ("is the product good?"). Teams with a contract diagnose quality problems in hours instead of weeks.

2. The contract specifies five categories of data: structured traces per CONTEXT layer, token counts per component, latency per stage, complete assembled context at inference time, and tool call results with memory load details. If the harness doesn't expose these, evaluations are flying blind.

3. Write the contract before the harness, not after. Retrofitting observability into a harness that wasn't designed for it costs 3-5x more than building it in from the start. The contract makes observability a first-class architectural requirement.