What It Costs, What It Returns

01Where Episode 04 left us

Five Paradoxes Every PM Must Hold turned each major design choice into a condition.

Upgrade the model after the workflow eval passes. Remove a rail when the failure it contained remains covered. Share infrastructure where scale helps. Keep workflow judgment local. Treat a demo as evidence of possibility and pair it with a hardening plan.

Every condition has a cost.

Verification consumes model calls and engineering time. Human approval creates a queue. Durable sessions, traces, and replay need infrastructure. Open interfaces cost more to build than a vendor default. Workflow-specific rules duplicate some work. Removing old scaffolding requires another round of evaluation.

The economics are not separate from reliability. They are the same system viewed from finance.

02The budget meeting

Month four. The CFO opens the spend report.

Model prices fell. Why did our AI line rise?

The PM can explain the token bill. That is not the difficult part.

The difficult part is showing why the team also paid for traces, evals, review queues, tool controls, durable state, and a staff engineer who spent a quarter making failures reproducible.

A production agent is cheap to call and expensive to trust.

The business case cannot be “we bought cheaper intelligence.” It must show what the surrounding system changed:

• FailuresFewer consequential failures.
• RecoveryFaster recovery when something does go wrong.
• EffortLower human effort per completed workflow.
• DelegationMore work the company is willing to delegate.

That is the reliability dividend.

03The sixty-second version

A model produces a proposed answer or action.

The harness is the system around it that decides what context the model receives, which tools it may request, when a human must approve, what counts as complete, and what gets recorded for later review.

In invoice reconciliation, the model may propose that an invoice matches a payment. The harness decides whether the evidence is sufficient, requests the invoice tool to update the record, checks that the update happened, and prevents the workflow from finishing while invoices remain unprocessed.

That surrounding system creates costs the model invoice does not show:

• Ground truthExamples and rules showing what the correct outcome should be.
• EvalsRepeatable tests comparing agent behavior against that ground truth.
• TracesStructured records of context, decisions, tool calls, and effects inside a run.
• RuntimeInfrastructure that keeps sessions running, stores state, resumes after failure, and limits resources.
• Human reviewExpert judgment for cases the system should not resolve alone.

This episode prices those five things and the value they make possible.

04Why cheaper calls can produce a larger bill

A chat interaction may require one or two model calls.

An agent workflow may require many:

1. Interpret the task.
2. Retrieve current context.
3. Choose a tool.
4. Process the tool result.
5. Repair an invalid output.
6. Verify the effect.
7. Continue after partial failure.
8. Ask a second model or human to judge the result.
9. Compact context.
10. Repeat until the business outcome is complete.

The price of each call can fall while total consumption rises because the system performs more calls, carries more context, retries, branches, and verifies.

This is not automatically waste. The extra work may be why the result can be used.

A three-call workflow that returns an unverified answer and a twelve-call workflow that completes the task correctly are different products. Comparing them by token cost is like comparing delivery services by fuel consumed while ignoring whether the package arrived.

The useful unit is cost per acceptable completed workflow.

For invoice reconciliation, “acceptable completed” means the correct invoices were matched, every exception was assigned, no prohibited action occurred, and the processed count agrees with the source ledger. The model's final sentence is not the unit. The verified business outcome is.

The three consumption multipliers

Agent costs rise through three common multipliers.

• ContextInstructions, history, retrieved material, tool descriptions, and state travel across calls.
• IterationThe workflow retries, verifies, and continues until an external completion rule passes.
• ParallelismSeveral agents or candidate paths run at once, often with an orchestrator and evaluator around them.

A model pricing page reveals unit cost. It does not reveal which multipliers your harness applies.

The harness owns the main levers:

• RoutingWhich model handles each step.
• ContextWhat is included, and when cached state can be reused.
• RetriesHow many attempts are allowed.
• ParallelismWhen parallel attempts are justified.
• JudgingWhich outputs require another judge.
• StoppingWhen the workflow stops.

Cost control is therefore a control-plane responsibility, not only a procurement negotiation.

05The real cost sequence

A common illustration shows a three-year curve: Year 1 front-load, Year 2 plateau, Year 3 return.

That shape can happen. The calendar is not universal.

A better model is a sequence of three phases.

Phase 1 — Foundation

The team builds what must exist before reliable throughput becomes visible: workflow definition, ground truth, trace coverage, tool and permission boundaries, completion evidence, human-review design, versioning and rollback.

The work is easy to call overhead because it does not immediately increase automated volume.

Phase 2 — Stabilization

Failures become reproducible. Incident classes fall. Review and recovery become faster.

This phase is politically difficult because much of the return is avoided loss. Finance sees infrastructure spend. It does not see the incidents that never reached a customer.

Phase 3 — Expansion

The same controls support more traffic, a higher autonomy level, or another workflow.

This is where compounding becomes visible. The second workflow does not need a new observability stack, identity integration, or release process from zero.

The invoice program looks different in each phase.

The phases describe how value appears. They do not prescribe a three-year calendar.

06Five cost centers the token bill hides

The order changes by workflow, scale, and what the organization already owns. Use the list as a complete budget, not a universal ranking.

Tokens, tools, compute, and runtime fees remain real. At fleet scale they can dominate. A token forecast alone is still not an agent budget.

07Cost center 1 — Ground truth and evals

Before a team can measure an agent, it must decide what acceptable work means.

Consider a partial payment. The invoice is for $10,000 and the ledger shows $8,000. Should the agent mark it paid, leave it pending, or create an exception?

The model can propose an answer. Ground truth records the organization's approved answer and the conditions behind it. An eval reruns the case after a prompt, model, tool, or policy change and checks whether the system still behaves correctly.

Writing that answer requires finance-domain judgment. Eval cost is not mainly the cost of running test code. It is the cost of making tacit business rules explicit.

For reconciliation, the full definition may include:

• MatchCorrect invoice-payment match.
• Edge casesCorrect handling of partial and duplicate payments.
• IsolationNo cross-tenant access.
• ExceptionsEvery exception assigned a valid state.
• ReconciliationFinal count reconciled against source records.
• EscalationCorrect escalation when evidence is insufficient.

Someone must label cases, resolve ambiguity, write deterministic checks, define rubrics, create holdouts, and maintain the set as policy changes.

Why eval work is easy to underfund

Features create visible output. Ground truth looks like a spreadsheet until a silent regression reaches production.

Budget eval authoring beside feature development, not as post-launch QA.

Each high-value workflow needs:

1. An initial golden set.
2. Named domain reviewers.
3. A regression suite in CI.
4. A production sampling plan.
5. A disagreement review.
6. A maintenance owner.

Deep measurement methodology belongs in AI Evals. The harness budget must still pay for it.

08Cost center 2 — Senior attention

Harness work consumes people with scarce context.

The staff engineer understands architecture, failure modes, and runtime trade-offs. The PM translates among user outcome, domain rules, security, finance, and engineering. Domain experts resolve cases a generic annotation team cannot judge.

Their time appears as opportunity cost: features not shipped, discovery delayed, reliability work postponed, senior reviewers pulled into calibration.

Name the capacity explicitly.

0.5 staff engineer for 16 weeks
0.5 product manager for 12 weeks
2 domain reviewers, 4 hours per week
Security and legal review at two release gates

The estimate may be wrong. An unnamed estimate is guaranteed to be missing.

09Cost center 3 — Observability and replay

Three terms are easy to blur.

• TraceRecords what happened during the run.
• EvalJudges whether that behavior was acceptable.
• ReplayRecreates enough of the run to test a fix.

If an invoice was marked paid incorrectly, the trace shows the context and tool call. The eval states that the result violated the partial-payment rule. Replay lets the team run the corrected harness against the same case before shipping.

You need all three. A trace without judgment is a detailed mystery. An eval without a trace can say “wrong” without explaining why. An incident that cannot be replayed is expensive to fix with confidence.

A useful agent trace may include:

• IdentityTask and tenant identity.
• VersionsModel and harness version.
• ProvenanceWhere each piece of context came from.
• CallsTool calls and normalized arguments.
• ApprovalsApproval decisions.
• ResultsTool result metadata.
• RepairsRetries and validation errors.
• EffectsState changes.
• CostLatency and token use.
• CompletionCompletion evidence.

The organization must store, query, redact, retain, and control access to that evidence.

Measure the return as:

1. Time to identify first divergence.
2. Percentage of incidents reproducible from stored evidence.
3. Time from production failure to regression case.

Observability is useful when those numbers improve, not when the dashboard count grows.

10Cost center 4 — Human review

“Human in the loop” sounds like a safety statement. At production volume, it is a queueing system.

Suppose 10,000 sessions run each day. If 3% require review, the queue receives 300 cases. At five minutes each, that is 1,500 minutes — 25 person-hours a day.

The review budget includes:

• InterfaceA reviewer interface with the correct trace context.
• RoutingRouting by expertise.
• Service levelA target, plus escalation and timeout behavior.
• CalibrationReviewer calibration and disagreement resolution.
• CoverageCover outside business hours.

The capacity model is simple.

• N — daily sessions.
• r — escalation rate.
• t — average review minutes.

A model improvement can lower r. Wider deployment raises N. Harder cases raise t. The review system must be priced against all three.

Human judgment does not necessarily disappear as models improve. It moves toward rarer and more consequential cases.

11Cost center 5 — Versioning and migration

A production harness changes on several clocks: model releases and retirements, prompt and skill updates, tool schema changes, policy changes, runtime upgrades, eval threshold changes, and new customer or regulatory requirements.

Without version discipline, the team cannot answer:

1. Which prompt and model produced this action?
2. Which policy version approved it?
3. Which tool contract was active?
4. Did the eval pass before release?
5. Can the prior behavior be restored?

Every meaningful harness artifact needs:

• VersionAn identifier that appears in the trace.
• OwnerA named person accountable for it.
• ReasonWhy the change was made.
• EvidenceThe eval result before release.
• RolloutA staged plan and a rollback path.
• RetirementA condition for removal when the artifact is temporary.

Episode 08 shows why deletion is part of the discipline, not a cleanup phase.

12Runtime cost is separate from model cost

Even an open harness needs somewhere to run.

Runtime includes the operational infrastructure for durable sessions, queues and workers, checkpoints, multi-tenancy, deployment, scaling, secrets integration, trace transport, and recovery.

A team can own workflow logic and evals while renting the runtime beneath them. That hybrid posture is often sensible.

Budget the runtime explicitly.

If the budget has tokens but no runtime line, it is incomplete.

13The reliability dividend

Productivity asks how much faster a person or team completes work.

The reliability dividend asks how the system changed the expected cost of being wrong, and how much more work the organization can now delegate.

“Permitted volume” is the amount of real work the organization is willing to place under agent control.

A company may have 100,000 eligible support cases but initially allow the agent to handle only 10,000. The limit may have nothing to do with model capacity. Legal, operations, or finance may not yet trust the system to detect and recover from mistakes.

If verification and rollback make failures cheaper, the company can raise that limit. The model handles more work because the organization's exposure changed.

The symbols below let a team insert its own numbers. Readers who do not need the equation can skip to the worked example without losing the argument.

• N — permitted volume.
• ps — successful completion rate.
• vs — value per successful completion.
• Crun — model, tool, runtime, and observability cost.
• Creview — human-review cost.
• pf — consequential failure rate.
• Lf — average loss per consequential failure.

The harness can improve value in four ways:

1. Raise ps through better context, retry, and verification.
2. Lower pf by catching invalid paths before release.
3. Lower Lf through containment, rollback, and recovery.
4. Raise N because the organization trusts the workflow with more volume.

The fourth is the reliability dividend people miss. The organization delegates more because the downside is bounded, visible, and recoverable.

14Autonomy is a loss-tolerance decision

An autonomy limit is a loss-tolerance decision expressed as product scope.

A team may let the agent draft every refund recommendation but issue only low-value, reversible refunds that pass deterministic policy. High-value or ambiguous cases go to review.

As evidence, rollback, and escalation improve, the issuing boundary can move.

The model may be unchanged. The organization delegates more because the consequences of error have changed.

15A worked example

Consider a support workflow with illustrative monthly assumptions:

• Eligible100,000 cases.
• Value$3 per correctly automated case.
• Review$4 human cost per reviewed case.
• Loss$30 average loss per consequential automated failure.

Read the example in this order:

1. How much work is the agent allowed to handle?
2. How many cases finish correctly?
3. How many need human review?
4. How often does a costly failure escape?
5. What remains after those costs?

The numbers do not claim that stronger controls always create this result. They show how a more expensive run can produce a more valuable system.

System A — weak controls

The organization permits 10,000 cases because failures are difficult to detect. Success rate 90%, consequential failure rate 2%, review rate 10%, run cost $0.20 per case.

Successful value = 10,000 × 0.90 × $3   = $27,000
Run cost         = 10,000 × $0.20        = $2,000
Review cost      = 10,000 × 0.10 × $4  = $4,000
Failure loss     = 10,000 × 0.02 × $30 = $6,000
Net value                                  = $15,000

System B — stronger controls

Verification, routing, and rollback improve. The organization permits 40,000 cases. Success rate 94%, consequential failure rate 0.5%, review rate 5%, run cost $0.28 per case because the system verifies more.

Successful value = 40,000 × 0.94 × $3    = $112,800
Run cost         = 40,000 × $0.28         = $11,200
Review cost      = 40,000 × 0.05 × $4   = $8,000
Failure loss     = 40,000 × 0.005 × $30 = $6,000
Net value                                    = $87,600

System B costs more per attempted case. Finance still prefers it under these assumptions because permitted volume rose fourfold, consequential failure fell, review became more selective, and expected failure loss stayed flat while useful work grew.

This is the pattern to test with your own data.

The cheapest run is not always the cheapest useful outcome.

16Named evidence, used carefully

Public case studies show what is possible. They do not isolate harness causality unless the experiment holds other variables fixed.

Rakuten

Rakuten reported that teams using Claude Code reduced average feature time to market from 24 working days to 5, a 79% reduction. It also reported a seven-hour autonomous coding task and 99.9% accuracy on a complex code modification in an open-source codebase.

The useful lesson is not that every team will get those numbers. The value came from a changed development workflow: parallel sessions, repository context, tool use, and sustained work around the model.

Budget the workflow redesign and review process, not only model tokens.

Azure SRE Agent

Microsoft reported that Azure App Service reduced time to mitigation for one live-site incident class from a 40.5-hour human-only average to 3 minutes with Azure SRE Agent.

The lesson is about allowed scope. Incident-response value appears when the organization lets the agent act across a defined path with the required diagnostics, tools, and controls.

Neither case proves that a harness alone caused the full result. Both show why the unit of value is the completed workflow, not the isolated model call.

17Quality now outranks raw cost

LangChain's 2026 State of Agent Engineering surveyed more than 1,300 practitioners. Roughly a third cited quality as the top production barrier, covering accuracy, relevance, consistency, tone, and policy adherence. Cost was cited less frequently than in prior surveys.

Cost still matters. The economic bottleneck moved.

Cheaper models made experimentation easier. The remaining value depends on making workflows dependable enough to expand.

Finance should ask two questions together:

1. What does one acceptable completed workflow cost?
2. What prevents the organization from delegating more of them?

A program that optimizes the first while ignoring the second can become cheap and irrelevant.

18What to rent and what to own

A restaurant may rent its building, payment terminal, and delivery software. It should not outsource what its food should taste like.

An agent team can rent models, sessions, execution, and trace storage. It should retain its definition of good, approval rules, workflow knowledge, and failure taxonomy. Those encode why the product is correct for this organization rather than merely operational.

Rent what improves through vendor scale

Often sensible to buy or reuse: model APIs, durable session infrastructure, sandbox provisioning, trace transport and storage, generic deployment and scaling, and common tool and protocol libraries.

Own what depends on your circumstances

Keep control of workflow definition, domain ground truth, eval cases and thresholds, authority and approval policy, escalation paths, failure taxonomy, and user and tenant context rules.

The hybrid is often correct

Rent operations. Own meaning.

A team may use a managed runtime while storing prompts, tool definitions, evals, policy, and trace exports in systems it controls.

The vendor dependency remains. Local judgment becomes easier to move.

19The lock-in test

For each harness cluster, ask:

Lock-in is not automatically bad. A closed product may be right for a commodity workflow.

The cost appears when the workflow becomes differentiating, regulated, or difficult to migrate after years of accumulated ground truth.

Price the exit before you need it.

20When to stop investing

Harness work can overshoot.

The twelfth verifier may cost as much as the second and improve little. A workaround may be close to becoming native. More depth on a mature workflow may produce less value than bringing another workflow onto the existing foundation.

Use three questions.

Is the eval curve still moving?

If several changes produced no meaningful improvement on representative and holdout cases, the team may be on the flat part of the curve.

Is the component compensation or institution?

If a provider is absorbing the generic capability, keep the wrapper thin. Do not build a permanent team around a temporary gap.

Would the next dollar buy depth or coverage?

A fraction of reliability on one low-risk path may be less valuable than applying proven controls to another painful workflow.

Stopping is capital allocation. A harness that only grows is accumulating, not compounding.

21Signs the program has earned expansion

A precise break-even quarter depends on workflow and starting point. Operating signals often appear before finance can isolate the full return.

• Prompt churn fallsThe team stops treating every failure as a wording problem and fixes schemas, state, tools, policy, or evals at the correct layer.
• Evals catch regressionsFailure cost moves left in the lifecycle, where correction is cheaper.
• Review turns selectiveReviewers handle genuinely ambiguous cases rather than rubber-stamping routine actions.
• Finance asks about expansionThe conversation changes from “should we continue?” to “which workflow comes next?”
• Autonomy expands with evidenceThe agent receives a new tool, limit, or traffic allocation because evals, rollback, and escalation support the change.

22The monthly value review

Do not build another dashboard unless these numbers are unavailable in existing finance, operations, and product systems.

The purpose is a monthly decision, not a visualization.

Read the review from outcome to cost:

1. How much work was permitted?
2. How much finished acceptably?
3. What failed, and what did failure cost?
4. How much human attention was required?
5. What did each acceptable completion cost?
6. Has the workflow earned expansion?

Use six lines:

Each line maps directly to the value equation. That is what prevents finance from seeing only token spend and prevents product from presenting only automation volume.

It does not need a new BI surface. A one-page monthly worksheet is enough.

The meeting ends with one of three decisions:

• ExpandIncrease volume, autonomy, or workflow coverage because reliability and economics meet the gate.
• HoldContinue while a named gap is closed.
• NarrowReduce scope because expected loss, review load, or unit cost exceeds the value produced.

If the review cannot lead to one of those decisions, remove it.

23Try this before Friday

Pick one workflow and fill in the value model with real numbers or explicit assumptions.

• VolumeHow many cases are eligible, and how many does the organization currently permit?
• SuccessWhat percentage reaches a validated business outcome?
• Run and reviewInclude model, tool, runtime, observability, and human review.
• FailureEstimate consequential failure rate and average loss. Use a range if the evidence is weak.
• Expansion gateWhat evidence would allow the organization to double permitted volume?

If the answer is “a better model,” name the failure the model must remove. If the answer is “more trust,” translate trust into eval coverage, rollback, escalation, and observed performance.

24What to remember

The model bill is one part of an agent's economics.

A production harness costs money because it turns a model response into a controlled business workflow. It supplies evidence, recovery, permissions, review, and a record of what happened.

That investment returns value in two ways:

1. Fewer and cheaper failures.
2. More work the organization is willing to delegate.

Measure both.

A system that lowers token cost while reducing successful completion is not cheaper. A system that costs more per attempt but safely handles four times the volume may be the better investment.

The next step is to replace assumptions with evidence from one workflow. Episode 06 is the audit that does that.