Evals as the New PRD

The spec that wasn’t a spec

A B2B PM ships a customer-support copilot in October 2025. The PRD is sixteen pages. It has personas, jobs-to-be-done, a north-star metric, three KPIs, a launch plan, and a one-page section called Quality Bar that reads: “responses must be helpful, accurate, and on-brand.” Engineering builds it. QA writes a test plan with thirty-two manual test cases. The feature ships. Internal demo goes well. The launch metrics in week one look fine — deflection rate at 28%, CSAT at 4.3.

Six weeks later CSAT on AI-handled tickets has crashed to 3.6. Deflection is up to 33% — more tickets are being closed by the bot — but the support manager is being pinged daily by enterprise customers about responses that are technically correct and operationally useless. The PM pulls a sample of fifty post-deflection conversations. About forty of them are fine. Eight are weird. Two are reputational landmines.

Here is what’s actually happening. The model didn’t regress. The model is doing exactly what the PRD asked for — helpful, accurate, on-brand prose. The mental model of “good behaviour” inside the PM’s head — what helpful means when a customer is venting, what accurate means when the policy is ambiguous, what on-brand means when the user is using profanity — never made it into a measurable artefact. The PRD described intent in prose. Production exposes that prose to a thousand inputs the PM never imagined. Without an eval suite, the gap between intended behaviour and actual behaviour can only be discovered by customers.

The PM did not ship a buggy feature. The PM shipped a feature whose specification was ambiguous, and ambiguity in an AI feature is what we used to call a defect.

The central frame

That sentence is the whole post. Carry it into the next architecture review, the next sprint planning meeting, the next handoff with engineering. Watch what changes when “the PM is responsible for the eval suite the way they used to be responsible for the PRD” becomes the working assumption.

The shift is not that PRDs go away. The shift is that the prose PRD stops trying to be the source of truth on behaviour. Behaviour is encoded in input-output cases — hundreds to thousands of them — that are versioned, reviewed, and treated as the contract between product, engineering, and the model. The PRD becomes a navigation aid: here is the customer, here is the job, here is the strategy, here is the link to the eval suite where behaviour is actually defined.

What an eval actually is — in plain English

An eval is a test case for a probabilistic system. It has three parts. An input — a prompt, a tool call result, a user message, a document, whatever the model sees. An expected behaviour — not always a single correct answer; sometimes a graded rubric, sometimes a constraint (“must not contain a competitor’s product name”), sometimes a structural property (“must include a citation”). And a scoring function — a way to programmatically or semi-programmatically decide whether the output meets the expected behaviour.

The eval suite is the collection of those cases organised by the behaviour they measure. P0 cases — the things that absolutely cannot break — sit in a tier that runs on every change. P1 cases run nightly. P2 cases run weekly or pre-release. The structure looks like a test pyramid only because the dependencies of probabilistic systems happen to rhyme with deterministic ones.

The thing the SaaS-trained PM has to internalise: an eval is not a unit test for code. A unit test asks “did the function return the right value.” An eval asks “did the system exhibit the right behaviour on this realistic input.” Behaviour is the unit. The eval is the operational definition of what good behaviour means — not in the PM’s head, not in the PRD’s prose, but in cases the model is actually scored against. The prompt is iterative; the eval is the durable artefact. That is the asymmetry the smartest 2026 AI teams have already internalised.

Why the money is following

In February 2026 Braintrust raised $80M Series B led by Iconiq at an $800M post-money valuation. For a six-year-old company whose entire pitch is “evals infrastructure for AI products,” that’s not a niche developer-tools round. It is the capital market saying: the operational discipline that PMs need to do well is now its own category, and the budget to staff it is real.

The Braintrust customer base — across foundation labs, vertical AI startups, and enterprise platforms — is running hundreds to thousands of eval experiments per customer per month. That cadence is the giveaway. A team running eight experiments per week is a team using evals as an end-of-sprint check. A team running thousands of experiments per month is a team that has restructured their workflow around evals as the loop. The first team treats evals as QA. The second team treats evals as the product. The valuation is following the second team because the second team is the one that ships AI features without production CSAT crashes in week six.

The pattern is the same one we saw with Inference FinOps in L2-T15 — when a discipline becomes investible, it has stopped being optional. Cost discipline got its own category in 2025. Quality discipline got its own category in 2026. PMs who treat either as a “will do later” line item are operating on a 2024 mental model in a market that has moved on.

The three categories of evals — and why PMs conflate them

Capability evals ask: does the model do X on this input. Can it summarise this earnings call without losing the guidance number. Can it extract a structured object from this messy CSV. Can it call this tool with the right arguments. These are narrow, model-centric, and reusable across products. Most foundation lab benchmarks live here. Most prompt engineering work converges here.

Product evals ask: does the end-to-end user task complete in a way the user would call good. The user uploaded their tax forms — did the agent file accurately. The user asked for a refund — did the bot resolve it without escalation while preserving the relationship. The user requested a code change — did the implementation pass tests AND match the codebase’s style. These are wider, system-centric, and almost never reusable across products because they encode the specific workflow your customer is paying for.

Safety evals ask: does the model violate boundaries it shouldn’t. PII leakage. Competitor mention. Tone breach. Jailbreak resistance. Hallucinated citation. Off-policy advice in a regulated domain. These are the cases where one bad outcome is worse than a thousand mediocre ones — the asymmetry that makes safety evals require their own treatment, their own scoring, and almost always a human-in-the-loop sampling layer.

The trap PMs fall into is conflating these three. A prompt engineer reports a 4% capability eval improvement and the PM reads it as a 4% product improvement. A safety eval pass rate of 99.2% sounds like a product-launch-ready number until you do the math on how many monthly conversations your product handles and realise 0.8% is a daily reputational incident. The eval shape has to match the question being asked, and the PM is the only person on the team responsible for asking the right question.

HHH — the operational quality dimensions

Anthropic’s research framing of HHH — Helpful, Harmless, Honest — is the cleanest taxonomy a PM can use to structure a product eval suite from a blank page. It works because each dimension is operationalisable as a distinct eval class with a distinct scoring function.

Helpful asks: did the response actually move the user forward on the job they hired the product for. This is rubric-graded by humans or by an LLM-as-judge calibrated against humans. Helpful is a product question, not a model question — a technically correct answer that ignores the user’s actual intent fails this dimension hard.

Harmless asks: did the response avoid causing damage. This is the safety-eval domain — boundary tests, jailbreak resistance, regulated-domain compliance, brand-tone protection. Harmless is binary or near-binary at the case level — it either crossed the line or it didn’t — but the suite-level score is a rate.

Honest asks: did the response represent its own knowledge accurately. Did it cite when it should have cited. Did it hedge when it should have hedged. Did it refuse when it didn’t actually know. Honest is the dimension most teams skip because it’s the hardest to score, and it is also the dimension where 2026 production failures have done the most damage to AI feature credibility.

The mistake is to treat HHH as three columns in a spreadsheet. They’re three suites with three release gates. A new model passes Helpful at 92% but Honest at 76%. A prompt change improves Harmless at the cost of Helpful. The HHH frame forces the PM to make the trade-off explicit instead of letting it leak into production unnoticed.

The Karpathy Loop — when evals become an optimisation target

In March 2026, Andrej Karpathy released Autoresearch — an open-source agent loop designed to hill-climb against an eval set without human intervention. It picked up over 42,000 GitHub stars in under a month. The headline isn’t the star count. The headline is what happened next.

Tobi Lutke pointed Autoresearch at Shopify’s templating engine. Over an unsupervised run, the agent produced 93 automated commits and a 53% improvement in rendering speed against the eval suite. The Shopify engineering team did not write those commits. They wrote the eval suite. The eval suite did the rest.

Aakash Gupta’s PM Guide to Autoresearch on March 19 2026 captured the shift in one sentence: Fortune called this “The Karpathy Loop.” The whole loop reduces to: define the success metric as a runnable eval, point an agent at the codebase, let the agent propose-test-keep-or-discard against the eval, repeat until the eval stops moving. The PM’s job in this loop is not to write code or to write prompts. The PM’s job is to write the eval that defines what winning looks like, with enough resolution that the agent can hill-climb on it without going off a cliff.

Read that again. The eval suite is the optimisation target. If the eval suite is wrong — incomplete, gameable, biased toward the cases the team imagined — the agent will hill-climb the model right off the cliff that the eval failed to anchor. Every PM who ships an AI feature in 2026 is now responsible for the integrity of an artefact that automated systems can optimise against. That responsibility didn’t exist in 2023. It is the single most underrated PM skill of this market cycle.

The LangChain Better Harness recipe — the operational protocol

On April 8 2026 LangChain published “Better Harness — A Recipe for Harness Hill-Climbing with Evals”. It is the most practical operational document a 2026 AI PM can read. The full recipe is worth reading end-to-end, but the three load-bearing moves are these.

Trace mining

You do not invent eval cases at a whiteboard. You mine them out of production traces — real user inputs, real model outputs, real failure patterns. The eval set you imagine is biased toward the inputs you can imagine. The eval set your traces give you is biased toward the inputs your users actually send. The first one tells you the team is creative. The second one tells you the product is alive.

Holdout sets

You do not score the model on the same eval set you optimised against. You hold out a representative slice — ideally drawn from a different time window, a different cohort, a different set of edge cases — and you only look at it on release decisions. This is the move that prevents eval-set overfit, which is the AI equivalent of teaching to the test, and it is the move that 80% of teams skip because the discipline cost is real.

Regression prevention

Every change — prompt, model, tool config, retrieval setting — is scored not just for what it improved but for what it might have broken. A 3% gain on Helpful that costs 5% on Harmless is a regression, not a release. The harness is the discipline that catches that trade-off before it ships, instead of after a customer escalation makes it visible.

The PM-facing summary of the recipe: trace-mined cases, separate holdout, regression-aware scoring. If the eval suite the team is using doesn’t have all three, the team is hill-climbing on a hill they haven’t measured.

The workflow shift

The old PM workflow was: write PRD → engineering builds → QA tests → ship → monitor. The PRD was the spec. QA was the verifier. Each was a different artefact owned by a different role.

The new workflow is: write eval suite → ship to eval suite → iterate on prompts/model/tools/retrieval against the suite → monitor production traces → mine traces back into the eval suite → repeat. The eval suite is the spec, the verifier, AND the optimisation target. It is a single artefact owned by the PM as the durable representation of what the product is trying to do.

The PRD survives — but its job changes. It becomes a thin overview document that captures customer, job, strategy, success metric, and a link. The link points at the eval suite. Anyone who wants to know what “good” means for this feature opens the eval suite, not the PRD. Engineering reviews the eval suite at design time the way they used to review wireframes. Design reviews the eval suite for moments where behaviour is going to feel wrong even when it scores right. Leadership reviews the suite-level pass rates by HHH dimension at quarterly business reviews — because that is where the operational truth of the product lives.

This is what Aakash Gupta has been calling “Evals Are the New PRD” for most of 2026, and it’s what the convergence of 2026 practitioner writing — across LangChain, Anthropic’s prompt engineering team, Shreya Shankar’s research, the Braintrust customer base, and the Karpathy Loop — points at without quite saying outright. The PRD is not dying. The PRD is being demoted to a navigation document. The eval suite is taking its old job.

The eval shape by archetype

The product architecture choice from L2-T13 — augmentation, copilot, agent — determines the shape of the eval suite. Skip this section if you skipped that one; come back when you’ve made the architectural decision.

Augmentation features

The model adds a narrow capability to a deterministic flow (smart compose, autocomplete, summarisation in a sidebar). The eval shape is narrow and capability-flavoured. Hundreds of input-output cases per capability. Helpful and Honest dominate. Harmless is shallow because the surface area is small. Holdout sets can be small. The recipe rhymes with traditional ML evaluation.

Copilot features

The model proposes, the human accepts/rejects/edits (code copilots, draft-an-email, suggest-a-reply). The eval shape pivots toward suggestion quality plus acceptance rate. You score not just whether the suggestion was good in isolation but whether a representative user would have accepted it as-is, edited it lightly, or thrown it out. The acceptance-rate metric pairs with the Adoption × Measurement value model from L1-T10 — a copilot with high suggestion quality and low acceptance is failing on adoption, and the eval suite has to surface both signals.

Agent features

The model takes multi-step actions in the world (book the trip, file the ticket, refactor the codebase). The eval shape is multi-turn task-completion. You score the trajectory, not just the endpoint. The AI Agent Autonomy Rate metric from L1-T03 lives here — the % of tasks completed without human escalation, weighted by task value and risk. Safety evals carry disproportionate weight because the cost of one bad agent action is bigger than the cost of one bad augmentation suggestion. Holdout sets need to span time, cohort, AND task type, because agents over-fit to task distributions in ways that augmentations don’t.

The same prose PRD can describe all three archetypes. The eval suite cannot. The shape of the eval suite IS the architectural commitment.

How evals pair with FinOps — the quality layer to L2-T15’s cost layer

L2-T15 was Inference FinOps — the cost-optimisation operational discipline. This post is the mirror image — the quality operational discipline. In production, every cost-optimisation lever pulls against quality, and every cost lever pull is unsafe without evals as the safety net.

Model swap from a frontier model to a mid-tier model — pre-pull eval comparison required. Prompt distillation from 800 tokens to 200 tokens — pre-pull eval comparison required. Adding a caching layer for retrieval — pre-pull eval comparison required. Routing rule change that sends 60% of traffic to a smaller model — pre-pull AND post-pull comparison required, because production traffic distribution will not perfectly match staging.

The four traps

If the central frame is the diagnosis, here are the four habits that quietly turn a well-intentioned eval initiative into theatre. Each one looks responsible. Each one ships failure modes the team will only see after a customer does.