Three Small Changes, Dramatic Outcomes

01Where Episode 02 left us

Inside a Production Agent Harness gave us five decisions around every model call.

Identity sets the operating contract. Memory policy chooses what the model sees. Orchestration decides what happens next. Interception checks the crossings. Observability and evals record and judge the run.

The reconciliation team can now draw its system. The map shows where the holes are.

The next question is practical: what can the team change this sprint?

02The sprint that changed no model

The team has spent a quarter comparing models. The invoice agent still fails in three familiar ways.

It repeats a malformed request without learning from the error. It returns fields the downstream system cannot use. It picks the wrong tool when several names sound alike.

On Monday, the team freezes the model.

Validation errors now return to the next attempt. Outputs must follow an agreed shape. The agent sees only the tools needed for the current step.

By Friday, the lesson is not that three tricks worked. It is why they worked.

Each change removed a decision the model kept getting wrong.

Reliability improves when the system replaces a vague choice with a specific signal or constraint.

03The whole pattern in one table

No code is required to understand the production loop.

The model proposes. Reliability comes from the decisions before and after that proposal.

04Pattern 1: retry with evidence

A naive retry says, “Try again.”

The model receives the same task, context, and uncertainty. A different answer may appear, but the second attempt has no better reason to succeed.

A useful retry explains what failed.

The second version supplies the missing information: field, observed value, expected rule, and correction scope.

• InformationReturn the exact validation error.
• StrategyAsk for a different method after repeated failure.
• ContextRestart with a clean view and durable progress.
• AuthorityEscalate to another model, tool, or human.

If none changes, the retry is repetition.

One validator cannot do all three jobs.

The key metric is not retry count. It is the percentage of failed first attempts that recover without creating a wrong business outcome.

05Pattern 2: constrain the output space

Teams often treat structured output as a developer convenience. The product value is a smaller failure space.

Free text lets the model return anything it can express. A schema, which is an agreed description of allowed fields and values, limits output to shapes the program knows how to handle.

Consider the invoice output.

A weak instruction such as “return JSON” may still produce:

• Amount"$42.50"
• Due"end of August"
• Status"awaiting payment"
• ExtraAn invented explanation field the caller ignores.

The response looks sensible to a human. The next system may fail or silently discard information.

A schema checks shape. An eval checks meaning. A completion rule checks the job.

OpenAI’s current documentation makes a useful distinction. JSON mode returns valid JSON. Structured Outputs with strict mode follow a supported supplied schema. Your application should still validate semantic rules and handle refusals, empty results, timeouts, and unsupported schema features.

A schema is a contract with every downstream consumer, not only the model. When it changes:

1. Version the new shape.
2. Identify affected consumers.
3. Test old and new cases.
4. Plan compatibility or migration.
5. Monitor empty, refused, and semantically invalid outputs.

A field added casually today becomes a silent mismatch three systems later.

Track schema-valid output and semantic correctness separately. A perfect shape can carry a wrong answer.

06Pattern 3: narrow the decision surface

Give an agent forty tools and it must distinguish forty operations before every action.

Some tools are irrelevant. Some overlap. Some use broad names such as query_data or update_record. The model must infer what they mean and whether the consequence is read, draft, or execute.

The solution is not always fewer total capabilities. It is fewer visible choices for this step.

A precise name is part of the control surface.

1. Delete tools with no distinct job.
2. Rename vague tools to exact actions.
3. Reveal specialist tools only when the task needs them.

The third uses progressive disclosure: the system keeps broad capability but shows the model a small, relevant set now.

A reconciliation skill may expose invoice lookup, payment lookup, exception drafting, and completion checks. A supplier-onboarding skill exposes another set.

A rare escalation tool may be essential. Review each tool on four dimensions.

The decision may be keep, rename, tighten, hide behind a skill, require approval, merge, or remove. The goal is not the shortest list. It is the clearest set that covers the workflow.

Near-neighbor tools fail at their boundary. Descriptions should include negative examples.

Use get_invoice_by_id when an exact invoice ID is known. Do not use it for vendor search or status lists.

The model now knows the operation and its edge.

Track correct-tool selection and steps per validated completion. Fewer calls are useful only when completion stays correct.

07Pattern 4: verify before exit

The invoice failure in Episode 01 did not involve malformed output or the wrong tool. The run stopped before the workflow ended.

Replace model confidence with an external checklist.

The model may propose completion. The harness checks the business state.

This is feedback at the job level:

• Output feedback“This field is invalid.”
• Completion feedback“This outcome is unfinished.”

Anthropic’s long-running-agent work uses feature inventories, progress files, startup scripts, and version history so new sessions can reconstruct what remains. LangChain’s Deep Agents work uses self-verification and pre-completion checks before exit.

A completion gate can create a loop. Define:

• BudgetTime and cost ceiling.
• AttemptsMaximum attempts without progress.
• RecordDurable progress record.
• EscalationWhere the run goes when it stalls.
• StopThe condition when the task cannot complete.

“Continue until done” is not a policy unless both done and stop are defined.

08The measured receipt

LangChain held the model fixed and changed the system prompt, tools, and middleware, its term for hooks around model and tool calls.

The work included trace-based error analysis, self-verification, pre-completion checks, environment context, and loop detection. Terminal Bench 2.0 performance moved from 52.8% to 66.5%, a 13.7-point increase.

The result does not prove every harness change helps every workflow. It supports a narrower claim: changing the control system can materially change measured behavior while the model stays fixed.

The method is more reusable than the benchmark:

1. Run a defined test set.
2. Read the failed traces.
3. Group failures by behavior.
4. Change one system surface.
5. Keep the change only if evaluation improves.

09The feedback loop

A production trace is a record of what happened. It becomes an eval only after the team states what should have happened.

A holdout set is a group of test cases the team does not use while designing the change. It checks whether the improvement works beyond the examples that inspired it.

Twenty well-labeled cases from real failures may be worth more than a thousand synthetic cases no user has produced.

10In practice: one sprint

Pick one high-volume model decision.

Do not ship all five changes at once if you cannot attribute the result. Start with the failure that appears most often or carries the largest consequence.

11Connecting the dots

The four patterns share one design principle. Let the model reason over the uncertain part. Do not make it improvise the contract around the uncertain part.

That principle connects reliability, cost, and autonomy.

A precise error reduces wasted retries. A schema protects downstream systems. A narrow tool set reduces wrong actions. A completion check lets the organization trust longer work. Each constraint may also add latency, cost, or blocked edge cases.

That trade-off is the bridge to Episode 04. The question is no longer whether constraints work. It is when they buy enough reliability, when they limit useful autonomy, and when a better model makes them unnecessary.