- 01. Why cost per API call is the wrong metric -- and why cost per successful task changes every decision about model selection, judge design, and human review routing.
- 02. How to build an eval budget as a portfolio: cheap broad coverage for routine cases, expensive high-fidelity review for ambiguous or high-stakes cases, and strategic sampling everywhere in between.
- 03. The trap that quietly doubles eval costs: forgetting that the evaluation system itself has its own economics that can outgrow the production system it measures.
The Story
A fintech company builds an AI assistant that answers customer questions about account balances, transaction history, and fee calculations. The team is thorough. They turn on full tracing, run three online LLM judges on every production trace, route 30% of traces to a human annotation queue with two reviewers per case, and run the full eval suite on every prompt change.
The system works beautifully. Quality is high. Regressions get caught early. Then the finance team asks a question no one anticipated: why is the evaluation infrastructure costing more than the production system it's evaluating?
Three judges on 200,000 daily traces costs more than the actual customer-facing responses. The annotation queue requires four full-time reviewers. Full-trace logging generates storage costs that grow linearly with traffic. Nothing was wrong. Everything was expensive. The team had built an evaluation system without ever treating it as a system with its own economics.
A second team takes the opposite approach. They optimize aggressively for cost per API call -- the cheapest model, the cheapest judge, fewer than 2% human review. Request-level costs look excellent. But the cheap model produces correct answers only 40% of the time on harder queries. Those failures create retry loops, escalation to human agents, and customer complaints. The cost per API call is low. The cost per actually-resolved customer issue is three times higher.
Braintrust's provider benchmarking makes this visible: the jump from an 8B model to a 70B model delivered a 16%+ accuracy gain, while 70B to 405B added only 2.87%. The relationship between model size and quality is not linear. The "cheapest per call" option is often the most expensive per successful outcome.
The Core Idea
Every evaluation choice spends something. A token. A judge call. A minute of reviewer attention. A CI pipeline delay. Individually, each cost looks small. Collectively, they form a budget that scales with traffic, and at production volume, that budget can quietly exceed the cost of the system being evaluated.
The metric that changes everything is not cost per API call. It's cost per successful task.
A system that costs $0.05 per call and succeeds 20% of the time costs $0.25 per successful task. A system that costs $0.20 per call and succeeds 90% of the time costs $0.22 per successful task. The cheapest request is not the cheapest outcome. You can't make this decision without knowing the success rate, and you can't know the success rate without the evaluation system you're budgeting.
That's the circularity at the heart of eval economics: you need evals to know what works, and you need a budget to run evals, and the budget depends on what works. The way out is to treat evaluation like a capital allocation problem.
| Model Tier | Cost/Call | Success Rate | Cost/Successful Task |
|---|---|---|---|
| Cheap (8B) | $0.05 | 20% | $0.25/task |
| Mid (70B) | $0.10 | 60% | $0.17/task |
| Strong (405B) | $0.20 | 90% | $0.22/task |
Where This Hits in Production
The cost-per-successful-task formula should be on every dashboard. Define it as: (model cost + retrieval/tool cost + evaluator cost + human-review cost) / number of tasks passing the acceptance bar. When this number trends up without a corresponding quality improvement, something in the system is degrading.
Tiered evaluation matches tiered models. If you route easy requests to a cheap model, you should also route easy evaluations to cheap judges. A deterministic format-compliance check costs near-zero. A semantic accuracy judge costs model tokens. A human expert review costs $3-10 per case.
Budget your human reviewers like infrastructure. How many cases per day can each reviewer handle? What's the queue depth at current escalation rates? If your annotation queue has a 72-hour backlog, your human review is decorative, not functional.
Common Mistake
Optimizing the cheapest layer while ignoring the rest.
Teams obsess over token price -- the most visible cost -- while ignoring retry rates, false positives from cheap judges, reviewer backlogs, and the cost of bad outputs reaching customers.
A 30% reduction in model cost that increases human escalations by 50% is a net loss. The metric that matters is cost per successful task, and that metric spans all four layers.
If any single layer accounts for more than 50% of total eval spend, it's probably over-allocated relative to its information value.Connecting the Dots
Topic 26 adds the economics layer that every previous topic assumed but never made explicit. CI/CD evals showed how to gate releases with automated evaluation. T26 asks: what does that gate cost to operate per PR, per day, per month? Production monitoring established continuous quality tracking. T26 asks: what does continuous mean when you're paying per evaluation?
The Pass^k reliability multiplier from Topic 16 changes the economics of agent evaluation in a way that's easy to underestimate. Evaluating a 10-step agent isn't 10x the cost of evaluating a single step — it's evaluating 10 steps plus the compounding interactions between them. The Pass^k math also determines sample sizes: detecting a 2% per-step reliability drop that compounds into a 19% end-to-end drop for a 10-step chain requires enough samples at each step to achieve statistical significance at the per-step level, not just the aggregate. Teams that budget eval costs by step count without accounting for the compounding interactions consistently underspend on the agents that need the most scrutiny.
The eval flywheel described the closed loop from production failures to harder eval suites. T26 adds the sustainability constraint. A flywheel that promotes 20 new cases per month and never prunes grows the suite — and the CI cost — indefinitely. For every case promoted, evaluate whether an existing case has become trivial. Replace, don't just accumulate.
Remember This
The right metric is cost per successful task, not cost per API call. A cheap model with a 20% success rate costs more per useful outcome than an expensive model with a 90% success rate.
Eval systems have four cost layers: production model, evaluators, human review, and infrastructure. Each has its own economic lever. Managing one layer while ignoring the others is local optimization.
Human review is scarce capital, not default workflow. Route it to the uncertainty zone -- cases where automated judges disagree, score near the threshold, or encounter novel patterns. A team that reviews 5% of traces strategically gets more signal than a team that reviews 30% uniformly.