Your First Eval Suite

The Story

Rechat's AI assistant was leaking raw database UUIDs into answers sent to real estate agents. Strings like a1b2c3d4-e5f6-7890, scattered through every otherwise-helpful response. The team had built sophisticated evaluations for conversational quality, helpfulness, and task completion. They had skipped the three-line check that would have caught every single leak.

A deterministic UUID-pattern assertion runs in two milliseconds and costs nothing. Rechat had to retroactively build hundreds of simple unit tests to course-correct. The sophisticated eval suite was built on a foundation that had a hole in the floor.

NurtureBoss, a property-management AI assistant, lived a quieter version of the same lesson. The system kept booking apartment tours on wrong days, scheduling walk-ins when offices were closed, and confusing time zones for multi-location properties. The team trusted vibe checks and the generic hallucination score from their eval tool. Both stayed green.

When the team finally extracted one hundred production traces and coded them by hand, date-handling failures accounted for the majority of all errors — invisible to generic quality scores, immediately obvious to the first human who read the actual traces. A fifty-case eval suite with one deterministic date check would have caught it before launch.

The Core Idea

This is the capstone of Level 1. Everything from Topics 1–9 comes together into a practical exercise: build the minimum viable eval suite. Not the comprehensive system you need in six months. The working system you need before launch.

Anthropic's evaluation guidance includes a number that should stop every PM: start with 20–50 simple tasks drawn from real failures. Not 500. Not a comprehensive matrix. Twenty to fifty. The reasoning is practical: early in a product's life, changes have large effect sizes. A prompt rewrite might improve accuracy by 15%. A retrieval fix might eliminate an entire failure category.

Teams with eval suites can upgrade models in days, while competitors without them face weeks of manual testing. The eval suite doesn't slow you down. It speeds you up. Every change becomes a measurable experiment instead of a prayer.

Why Starting Small Wins

Consider a Fortune 500 logistics company whose CTO mandated a massive enterprise-wide observability suite before launching a supply-chain routing agent. The team spent months integrating the APM tool, generating dashboards with generic metrics — helpfulness scores, toxicity scores, latency distributions. When users complained the routing was inefficient, the dashboards were green.

Everything looked healthy because they were measuring infrastructure, not domain quality. What they skipped: building 20–50 actual routing scenarios with correct expected routes, verified by logistics experts.

Enterprise teams often try to build the comprehensive system from day one — governance workflows, multi-tenant datasets, automated judge calibration, slice-level alerting. They spend months planning. Nothing ships. The product degrades in production with zero eval coverage while the eval team designs the perfect system.

Starting small produces a better eval system because production data teaches you what matters. Your 20 cases are a hypothesis about what breaks. Production tells you the truth. Two weeks of production traces reveal failure patterns your design phase never anticipated.

Where This Hits in Production

The PII data scrubbing tax. In regulated enterprise, those 50 traces from production contain PHI or PII. Before the PM can even run their MVP eval, they need a data-sanitization pipeline. The workaround: use functionally equivalent mock data for the first iteration, then backfill with sanitized production traces as the pipeline matures.

Procurement gridlock vs. notebooks. Enterprise teams often wait six months for security to approve an evaluation platform. The MVP eval can't wait for procurement. Run the first 50-trace eval in a local Jupyter notebook to prove value. Once the eval demonstrates impact, the case for procurement writes itself.

Multi-tenant baselines from day one. A 50-trace eval suite averaging 90% globally might mask 100% failure for Enterprise Tenant B. Include tenant_id metadata on every golden case from day one. The global average is a vanity metric. The per-tenant pass rate is the operational truth.

Connecting the Dots

Strip away all the complexity, and every eval reduces to exactly three things: data (inputs your product needs to handle), a task (the function that produces output), and scores (a number between 0 and 1 that says how well the task performed). Everything else is iteration on those three components.

The counter-argument deserves honest treatment. Everyone is already doing evals — they just don't call them evals. When you dogfood your product, analyze what feels off, and iterate based on feedback, that IS evaluation. The question isn't whether you evaluate, but when you can afford to be less rigorous and when you cannot.

Two cases where lighter evals work: your task is well-represented in model posttraining, or your team has deep domain expertise and religiously dogfoods. Everyone else needs formal structure. The moment your product's outputs require judgment to evaluate — tone, policy adherence, clinical accuracy — vibes stop scaling. That's the boundary.

In Practice: The One-Week Build

Monday–Tuesday: Pull 30 real cases and tag each with expected correct behavior. Write 4 automated checks — valid structure, no leaked personal data, no leaked internal IDs, proper citation.

Wednesday: Add one real-time safety filter that blocks dangerous outputs before the user sees them, with a graceful fallback message. Instrument every pipeline step with OpenTelemetry so when something fails, you see exactly which step broke.

Thursday: Set up one AI evaluator that checks the single most important quality dimension for your product. It runs in the background on 10% of live traffic, scoring each response pass/fail with an explanation.

Friday: Wire into CI/CD. If critical test cases fail, the deploy is blocked. No code change reaches production without passing the golden dataset checks first.

The best teams aren't running evals occasionally. They're running 12+ experiments per day. They have a morning ritual: pull up yesterday's production logs in standup, reconcile what they see with eval scores, find novel patterns, add them to the dataset, and grind on those failures all day. The eval suite isn't a gate they pass through on the way to shipping. It's the operating system they ship through.

Remember This

1. A minimum viable eval suite has six parts — golden dataset, code assertions, one safety guardrail, one async judge, traces, a comparison dashboard. Build all six before launch. Skip one and the others stop catching what they were built to catch.

2. Twenty cases beat five hundred. Your first dataset is a hypothesis. Production tells you what actually breaks — and what to add next.

3. A 100% pass rate is a smell, not a trophy. If everything passes, your eval is too easy. Add harder cases until 20–40% fail. That failure zone is where improvement happens.