Three Small Changes, Dramatic Outcomes

The Team That Fixed Their Agent Without Touching the Model

An engineer at one of the labs told me this over coffee last month: "We fixed reliability by removing three lines of code and adding two. Nothing touched the model."

His team had spent a quarter chasing a better model. They tried Opus, Gemini 2.5, a fine-tune of their own. Drift stayed. Then a new hire, on week two, rewrote the retry loop, turned on strict JSON mode, and cut the tool list from thirty-one to nine. Drift dropped by over fifty percent in a week. The model they ended up keeping was the one they started with.

Quick check on where we are before we go further. Episode 1 started at 2 AM and reframed the problem, the model is one part, and the real system is four layers standing side by side. Episode 2 opened those four layers into a street map; the harness got its own five-part anatomy because that is the layer you own. So the map is drawn. What has been missing until now is the answer to the obvious next question: what do I actually change on Monday?

This episode provides answers. Three small changes on the map you just built, each pinned to a specific cluster inside the harness, Interception, Memory Policy, Orchestration, plus one meta-move in Observability & Evals that turns the three into a compounding program instead of a one-week win. It is also the exact sprint the reconciliation team from Episode 2 runs the week after their whiteboard session, which is why you will recognize their invoices in every code block below. The fastest way to learn these patterns is to watch them land on the workflow that was failing at 2 AM.

One rule from Episode 1 still holds while you read: the user stays outside the stack. The request did not change, the task did not change, the user did not change. The moving part is always in the harness. A change you cannot locate on the map is a change you cannot make twice.

The idea in one box

Reliability is bought with small deterministic changes. A retry that learns, an output space that is bounded, a tool gate that is narrow, an artifact you can replay. None of them touch the model. And they compound only when the meta-move wires them into a flywheel that turns every production failure into a permanent eval.

Why Small Harness Changes Beat Big Model Upgrades

What are the highest-leverage harness moves a team can make this sprint?

Not this quarter. Not after the next offsite. This sprint. The changes I'll walk through are small enough to implement in two days, large enough to show in your eval dashboard by Friday. None require a new model. None require a platform migration. All are counterintuitive, which is why most teams neglect them.

And the evidence has caught up with the intuition. The May 2026 survey of 170+ agent projects measured exactly this: simple harness tweaks, better tool formats, sandbox changes, automated verification loops, delivered larger reliability gains than model upgrades in the same environments. The cheapest moves on the board are also the highest-return ones. That is rare in engineering.

Take it while it lasts.

Here's what most teams do when their agent produces a malformed output. The model returns broken JSON. The hThe harness sends the exact same prompt back, hoping for a different result., hoping for a different result.

On retry number two, it does the same thing. By retry three, the team logs a failure and pages someone.

This is the retry pattern almost every harness ships with on day one. It's also the wrong pattern.

The model did not fail because it was lazy. It failed because something specific broke. A missing field, a trailing comma, a string where a number was expected. When you resend the same prompt, you are asking the model to guess what went wrong with no new information.

The retry is a coin flip. Sometimes the coin lands the right way. Most times it fails, burning tokens and emptying your error budget.

The structured retry works differently. It treats the failure as a signal. It parses the error, extracts the violation, and feeds it back as context.

Same model. Same tools. Same temperature. Different information.

The augment function is the whole game. It writes something like: "Your previous response failed validation." The error was: expected 'amount' to be a number, got string '$42.50'. Return a corrected response that fixes only this field."

That is the structured retry. It cost two engineers one afternoon to build. In teams that deploy it, drift (the rate at which sessions fail) dropped by about sixty percent. Same model. Same prompt. New feedback loop.

Three specifics most teams miss when they go to implement this.

Cap the feedback window at the actual violation.   Do not dump the full failed output back into the retry prompt. Bloating tokens and confusing the model with irrelevant context. Feed only the broken field, the specific error, and the expected shape. "The amount field was a string." Return a number in cents." That is it.

Retry budget is separate from retry behavior.   Three retries is a sensible cap, but the retry logic on attempt one should be the same on attempt two. Do not escalate to a "bigger model" on retry three, that hides the harness bug under a more expensive model call. Fix the feedback, not the compute budget.

Log every retry with the error context you fed back.   This is your dataset. Working structured retries teach nothing. Failed structured retries reveal errors your augmentation prompt can't fix. Feed those findings back into the prompt template. This is how the retry loop compounds over months.

The same principle now has a higher-altitude expression, and it shipped while I was revising this episode. In June 2026, LangChain shipped rubric support in deepagents. You hand the agent an explicit, machine-checkable definition of done, and the harness forces the loop to continue until the rubric is satisfied.

It is the structured-retry idea lifted from output validation to goal validation, and it works for any agent, not just code generation. The underlying requirement does not change between altitudes: the harness must give the model a clear, checkable signal of progress or completion, or the loop has nothing to optimize against. A loop without a rubric is a treadmill.

The Ralph Loop lives next door. If structured retry is the Interception-cluster version of the retry pattern, the Ralph Loop is the Orchestration-cluster version. It catches the model's exit attempt rather than its malformed output attempt, checks progress against a completion goal, and restarts with a clean context and a progress file pointer.

Episode 2 named Ralph as an Orchestration pattern; Episode 8 treats it as a concrete example of the feedback flywheel. Here, the point is only that retry logic has two altitudes. Output-level, which is what structured retry does, and run-level, which is what Ralph and rubrics do. And they live in two different clusters and fail in two different ways.

Most PMs think of structured output as a polish layer. "Oh, that's what we'll turn on before shipping to Enterprise." They treat it like a designer treats a loading skeleton. It's a finishing touch.

This is exactly the wrong mental model.

Structured output is not a formatting feature. It is a reliability feature. When you tell the model "return JSON matching this schema" and the provider enforces strict mode, you are not making the output easier to parse. You are shrinking the set of things the model is allowed to say. Every token the model generates is schema-checked before leaving the model.

Invalid paths are pruned at generation time, not at parse time.

The difference is substantial.

In free-form generation, the model can produce any string. Most strings parse. A dangerous fraction look correct but silentAn `amount` field becomes a string with a dollar sign, a `date` becomes a human-readable phrase, an enum takes an unrecognized value.stream system doesn't recognize.

Your tests pass. Production fails.

With strict JSON schema enforcement, those drifts cannot happen. The model is only allowed to return something your code already knows how to read. It went from "can hand you literally any text" to "can only hand you a shape you have already agreed to accept." That is a much smaller room to fail in.

Three specifics separate a reliability schema from a formatting schema.

additionalProperties: False   Without this, the model can invent fields your downstream code will silently ignore. Insist on it. Every ignored field is a latent bug.

Use regex-bounded strings.   A date field typed as string is a hand grenade. A date field typed as string with pattern ^\d{4}-\d{2}-\d{2}$ is a contract. The regex is the contract.

Enums everywhere you can.   Any field with a small, finite set of valid values should be an enum, not a string. Free-form status: "paid" is a free-form status: "PAID" waiting to happen. Free-form strings introduce drift.

One more thing teams forget. The schema is not just a contract with the model. It is a contract with every downstream consumer of the model's output. When your schema changes, every parser, pipeline step, and report must change. Treat the schema like a public API. Version it.

Write a migration when it changes. Loose versioning hides costs for three months, then crushes in month four when new fields are silently dropped and nobody can reconstruct the timeline.

Give an agent forty tools, and it must check all forty each turn before acting. Most tools are wrong for this turn. A few are near-duplicates. A few sound right but behave subtly differently. The model often picks incorrectly.

Give the same agent eight carefully chosen tools, and its reliability increases. Its reliability increases, not because the model improved. It's because the choice surface became sharper.

Vercel removed roughly eighty percent of their agent's tools in a recent rewrite. The result: fewer steps per task, fewer tokens, faster completion. The agent did more with less because less was available.

This is the narrow-gate principle: the tools an agent can reach for are the tools that define its behavior, and every tool you add is a tax on every decision it makes.

Here's how most teams end up with forty tools. In sprint one, you add five. In sprint two, someone needs an edge case, so add three more. In sprint five, you have an internal tools library, so add fifteen for completeness.

In sprint twelve, a new team member sees the mess, proposes consolidation, and gets told "we don't have time for tool hygiene." In sprint twenty, your agent's accuracy is stuck, and you're blaming the model.

The fix isn't a platform migration. It's a spreadsheet. List every tool the agent exposes. For each, answer:

1. How often is this tool used? If used in less than five percent of sessions, it's a candidate for removal.
2. Does it overlap with another tool? search_documents
3. Is the tool name specific enough? Tools named query_data or run_analysis are magnets for misuse. Rename to what they actually do: get_invoice_by_id, list_overdue_invoices.
4. Does the input schema constrain the agent, or invite hallucination? Free-form string arguments invite hallucination. Typed, enum-bounded arguments do not.