Evals Are the New PRD

The Story

A fintech company builds an AI assistant that helps small-business owners understand their cash-flow projections. The product requirements document is thorough: the assistant should explain projections in plain language, flag unusual patterns, avoid giving specific financial advice, and cite the underlying data when making claims. The PM signs off. Engineering builds. QA reviews output samples manually. The team ships.

Three months later, the PM is frustrated. Support tickets mention that the assistant “sounds robotic.” The VP of Product wants to know why activation is flat. Engineering points to the eval suite — accuracy on financial calculations is 94%, citation completeness is 91%, advice-boundary compliance is 97%. By the team's own metrics, the system is performing well.

But the PRD said “explain projections in plain language.” Nobody ever operationalized “plain language.” It wasn't in the eval suite because nobody could agree on what it meant. The dimension that actually drove user satisfaction was the one dimension that existed only as a sentence in a Google Doc. The PRD described the aspiration. The eval suite described what the team actually measured. The product lived in the gap between them.

A second team at an insurance company does build comprehensive evals. They measure accuracy, safety, tone, completeness, and latency. But the eval suite lives in the ML team's repository. Product managers don't read it. They write quarterly OKRs around policyholder satisfaction and claim-resolution speed. When the ML team improves tone scores by 12%, nobody on the product side notices because they're tracking NPS through a different pipeline entirely. Both teams are right within their own measurement systems. Neither system talks to the other.

A third team takes a different path. Their product lead starts attending eval review meetings. She notices that 40% of the failure cases in the weekly annotation queue involve a use case the team never designed for — users asking the assistant to compare competitors, not just explain their own data. The PRD never mentioned competitive comparison because the original user research didn't surface it. But production traces showed it was already the second most common use case. The eval system wasn't just testing the product spec. It was rewriting it.

Three teams, three positions on the same spectrum. The first had a PRD but no eval system that matched it. The second had both but they didn't talk to each other. The third discovered that the eval system, properly connected to production reality, doesn't just enforce the product spec — it generates the next one.

The Core Idea

In traditional software, the product spec and the test suite serve different functions. The spec describes intent. The tests verify behavior. The two coexist because deterministic software does what it's told — if the logic is correct, the behavior matches the spec. The spec is authoritative. Tests are verification.

AI systems broke that clean separation. A prompt-based system doesn't execute logic — it generates behavior probabilistically, shaped by model weights, prompt instructions, retrieval context, and tool availability. The same input can produce different outputs on different days. “Does it work?” is no longer a yes/no question. It's a distribution question: does it work well enough, often enough, on the right dimensions, for the users who matter?

That distribution question can't be answered by a document. It can only be answered by a measurement system. Which means the operational definition of “success” migrates from the PRD to the eval system. The PRD still captures strategy, positioning, and vision. But the eval system carries the executable contract: what “good” means in production, right now, measured against real user behavior.

Going Deeper

Why the migration happens. The migration from PRD to eval system isn't a philosophical choice. It's a structural consequence of three properties of AI products. Non-determinism requires continuous measurement — a PRD that says “responses should be helpful and accurate” is a tautology when every team agrees with it but nobody can operationalize it without a scoring function. The scoring function is the operational definition. The product changes faster than documents can track — every prompt edit, model upgrade, or retrieval change shifts behavior. And users define the product in production — they find use cases the team never designed, visible only through production traces.

The three layers of the executable spec. Once you accept that the eval system carries the operational definition of success, it helps to see the system as three layers. The scoring contract is where “helpful,” “accurate,” “safe,” and “on-brand” get decomposed into measurable dimensions with explicit scales. The living dataset consists of golden datasets curated and continuously updated from production — the case law that interprets the scoring contract. The business bridge connects offline eval scores to business outcomes, translating “our tone score improved by 8 points” into “this correlated with a 3% lift in 7-day retention.”

Production traces as product discovery. This is the part most teams miss, and it's the most powerful implication. Traditional product discovery works upstream: user research, surveys, interviews, competitive analysis. That workflow still matters. But AI products generate an extraordinarily rich signal from production traces — what users actually ask, how the system responds, where conversations break down, which use cases cluster together. The product spec isn't written once upstream. It's continuously rewritten from evidence generated downstream.

Where This Hits in Practice

The team with a PRD and no eval alignment. Product, legal, support, and engineering all use the word “quality” but mean different things. Nobody operationalized the terms. The fix isn't more PRD detail — it's a scoring contract that forces every stakeholder to define their dimension numerically and accept a threshold.

The team with evals disconnected from business metrics. The ML team optimizes eval scores. The product team tracks business KPIs. Both improve on their own dashboard. Neither moves the customer outcome. The fix is connecting every AI-config change to a business metric, not just an internal quality score.

The team that isn't using traces for discovery. Product managers run user research cycles while the most detailed behavior data in the company sits in production traces, unexamined. The fix is making trace review a product activity — put PMs in the annotation queue rotation, surface pattern clusters in product review meetings.

In Practice: Building Evals as Your Product Spec

Step 1: Audit the gap. Take your current product requirements and list every quality dimension mentioned. “Accurate,” “helpful,” “safe,” “professional,” “fast.” Now list every dimension your eval suite actually measures. The gap between those two lists is your Spec Drift exposure. Every unmeasured dimension is governed by opinion, not evidence.

Step 2: Build the scoring contract as a shared artifact. For each dimension that matters, define: what it means numerically, what threshold constitutes “acceptable” versus “good,” who owns the definition, and what decision the score governs. This isn't an engineering document. It's a product document that happens to be executable.

Step 3: Connect offline scores to business outcomes. Pick your top three eval dimensions and establish correlations with business metrics — retention, resolution rate, NPS, conversion. If a dimension improves by 10 points and no business metric moves, either the dimension doesn't matter or the business metric is lagging.

Step 4: Make production traces a product input. Set up annotation queues. Staff them with product and engineering together. Review traces weekly. Look for: new use cases the team didn't design for, failure patterns that repeat across segments, and coverage gaps where the golden dataset doesn't represent actual usage.

Step 5: Establish the trace-to-spec feedback loop. Production traces → annotation → golden dataset updates → scoring contract revision → release criteria. Assign ownership at each handoff. Set a cadence: weekly annotation review, monthly dataset refresh, quarterly scoring contract review.

Step 6: Match formalization to maturity. Don't lock down scoring contracts for features still in exploration. Use rough pass/fail during discovery, dimensional rubrics during validation, full scored contracts with business bridges during scaling. Progressive formalization — not everything needs the full spec treatment on day one.

Step back and look at what happened to your role as you moved through this series. You started writing golden test cases by hand — individual examples of good and bad output, crafted one at a time. Then you built evaluation pipelines: scoring functions, annotation workflows, CI/CD gates, production monitors — systems that scale quality judgment beyond what any individual could review. Then you designed governance boundaries: progressive deployment gates, kill switches, red-team protocols — constraints that determine what autonomous systems are allowed to do. And now, here, you've arrived at something that subsumes all three. When the eval IS the PRD, you're not writing test cases, building pipelines, or even designing constraints. You're defining what “good” means at the organizational level, and the entire evaluation infrastructure — the scoring contracts, the living datasets, the business bridges, the trace-to-spec feedback loops — self-organizes around that intent. The eval system doesn't just test the product. It is the product specification, continuously rewritten from production evidence. That's the Intent Architecture framework from Topic 32 in its fullest form: the PM's role evolves from Example to System to Constraint to Intent, and T29 is where Intent becomes operational.

Remember This

Most teams that struggle with AI product quality don't have a measurement problem. They have an alignment problem: the written product definition says one thing, the measured product definition says another, and nobody notices the divergence until customer outcomes stall.

Evals as product spec doesn't mean abandoning PRDs. It means recognizing that in AI products, the evaluation system carries the operational definition of success. The PRD provides direction. The eval system provides ground truth. When they diverge, every team optimizes for their own dashboard while the customer experience falls through the gap.

The practical shift: treat the eval system as a product artifact, not just an engineering one. Put PMs in annotation queues. Connect eval scores to business metrics. Build golden datasets from production traces, not pre-launch imagination. Review the scoring contract quarterly with the same rigor you'd review the product roadmap.

The business plan tells you where you want to go. The P&L tells you where you actually are. Run the company on both.