- 01. The nine-day audit that diagnoses your harness before you touch anything. Including an Evals Onboarding Day and an Open-Harness Audit that most kits skip.
- 02. The five-rung harness maturity ladder. Where your team sits, and the next rung.
- 03. Four tickets every team should file before the next sprint. Exact wording.
- 04. The stakeholder script. Three translations of the same investment for engineering, design, and the exec team.
- 05. The numeric calibration target for your eval suite. When LLM-as-judge graduates from rough proxy to unattended primitive (>90% agreement with humans on a 50-case calibration sample).
The Monday Where the CEO Asked "So What Do We Actually Do?"
MThe reconciliation team's PM has survived budget review, paradox meetings, and whiteboard sessions, and now faces the question none of the framing answers: what does week one of the discipline actually look like, not just the heroic fix but the repeatable version?eatable version? The calendar is full.
The team is mid-sprint. Nobody has approved a platform migration, a new hire or a committee. Good. None of those are needed.
Most harness advice lives at thirty-thousand feet. This episode lives at your keyboard. You've read five episodes of framing. The four layers, the five clusters, the three patterns plus the flywheel, the six paradoxes, and the cost shape in Episode 5. What It Costs, What It Returns.
What you don't have yet is a plan for Monday morning. That's what this episode is.
No new theory. No new model. No committee. Nine days of work you can start the morning after you finish reading, producing four tickets your team can close before the next sprint ends. The user stays outside the kit. Every audit day below is about the stack that serves them, not about them.
Heath Ramsey at ServiceNow names the starting move: “Organizations that have successfully deployed agentic AI most often start by identifying a high-friction process” (InfoWorld, April 2026). Pick the workflow that hurts the most every Friday. That is the one to point an agent at. It's not the most demo-friendly one, but the one your users complain about. Anurag Gurtu at AIRRIVED puts it the other way: “Start with decisions, not demos. What doesn't work is treating agents like stateless chatbots or replacing humans overnight.”
The nine-day audit below assumes you have already picked the workflow. If not, that is Day 0. Pick the friction. The kit follows.
The idea in one boxYou already have a harness. It was built reactively, by five people, over a year, with no map. The kit is nine days of looking before touching: name what exists, read what fails, wire the flywheel, audit for lock-in. Diagnosis before surgery, every time.
The Central Question
What does a PM do this week to start building the harness their org actually needs?
Not after a platform migration. Not after the next hire. This week. With the team you have, the budget you have, and whatever time you can carve out of a calendar already full of things that feel more urgent.
The answer is a kit. Seven days of diagnostic work, one page of output, three tickets. The kit is small enough to fit a normal sprint and sharp enough to make the second sprint twice as productive.
Part 1, The Nine-Day Audit of Your Current Harness
Before you build, assess. A harness you don't understand is a harness you can't improve. Nine days, one activity each. Each day produces an artifact. A page, a spreadsheet, a short list. By Day 9, you have a diagnosis specific enough to act on.
Days 1–7 are the core audit; Days 8–9 are the two most kits skip. An Evals Onboarding Day and an Open-Harness Audit are included. Skip them and you ship a tactical sprint without the two primitives that make the sprint compound.
Map what's already there.
You already have a harness. You just haven't named it. Every team running an agent in production has harness components in the codebase. Retry logic, some logging, a prompt template, a tool registry, a validation layer – all built reactively over six to twelve months. Pull the agent's code open. For each of the eleven components from Episode 2, write two things: what exists and where it lives. If something doesn't exist, write "missing." Do not fix anything today. Just draw. The map is the first honest conversation your team has had about the agent in a year.
Artifact → One page. 11 rows. 2 columns.Read 50 production traces.
Focus on sessions, not requests. These are full conversations, from first turn to resolution. Look for the three failure signatures from Episode 1. Premature stop. The agent gives up on a task it could have finished. Infinite loop. The agent retries, retries, retries and never makes progress. Silent drift. The output looks right, but a field is wrong, a date is off, or an enum the downstream consumer can't handle. Tally which signature dominates. Infinite loops point at retry logic. Silent drift points at schema enforcement. Premature stops point at context or tool selection.
Artifact → Tally sheet. 3 rows. 1 number each.Interview the staff engineer and the on-call.
Allow thirty minutes for each. Ask two questions, in this order. First: "What do you quietly work around?" This surfaces hacks, manual overrides, and Slack threads where someone fixes a failed session by hand before the user notices. Workarounds are signals. They tell you where the harness has been patched around instead of fixed. Second: "If you had one sprint and nobody asked for a new feature, what would you fix?" The overlap between the two lists is your first ticket.
Artifact → Two bullet lists. One overlap section.Catalog tools the agent can call.
Pull the tool registry. For each tool, answer three questions from trace data. Not intuition. How often is it called? When called, does it produce a useful outcome? Is it duplicated or near-duplicated by another tool? Most teams find the same pattern: 20% of tools handle 80% of sessions. Another 20% are used in fewer than 5%. The remaining 60% are duplicates, or were added once and never removed. The bottom third is your cut list. This is how you build toward Ticket 3.
Two more inventories were added on Day 4 in 2026, and both take under an hour. Let's review the MCP server inventory. For each MCP server in the registry, is it controlled by your teaEvery MCP server is a portability and attack surface.d an attack surface. Audit the list with the same rigor as an IAM review. Let's review the agent-identity inventory. For each agent, does it act under a human's borrowed credentials, or does it hold its own principal, account, permissions, and audit trail, scoped per compartment? Claude Tag (June 2026) is the reference implementation of the second pattern. An agent running on borrowed credentials is an audit finding waiting for its auditor. Note which of yours are.
The MCP 2026-07-28 release candidate removes the protocol-level session and the Mcp-SessioAny MCP request is now routable to any server instance; routing moves to headers your gateway and load balancer can read.d balancer can read. If your tool needs state, pass an explicit handle, a basket_id, a browser_id, as an ordinary tool argument. Do not rely on a sticky session unless you are locked to an an older MCP version. During the audit, tag every MCP server with the version it targets and the state-handle strategy it uses; that tag indicates its readiness for horizontal scale.
The identity side pairs with this. Artifact → 3-column spreadsheet. Sort by frequency.
Find the eval gap.
Pull your eval suite. For each behavior you claim the agent handles, answer yes or no: Is there an eval that fails when this breaks? You will find gaps, and every team does. The gap you want to name is the one where production failures and eval coverage don't overlap. The agent breaks in production in a way your evals never catch. That's your eval debt. That's the behavior your harness is blind to.
One more row to check beforePractitioners shipping long-running agents in 2026 keep naming the same highest-leverage gap: efficient verification, plus human priors injected where the model is still weak. is still weak. If your eval-gap list includes “we have no machine-checkable definition of done for this workflow,” that is the gap to close first. Everything else on the list is measurement. That one is steering.
Artifact → 2-column list. Rows with "no" = eval debt.Draft the maturity diagnosis.
Step back. You have a component map, a failure tally, a staff engineer's wishlist, a tool audit, and an eval gap analysis. You have enough to place your team on the ladder (full ladder in Part 2). Write one paragraph: "We are at Rung [N]. The evidence is [specific artifact]. The next rung is [N+1]. The move that closes the gap is [specific component]." One paragraph. It's specific enough to read aloud to your engineering lead and have them nod, not argue.
Artifact → One paragraph. Specific. Defensible.Write the four tickets.
The audit compresses into four tickets (templates in Part 3). The highest-volume call from Day 4 becomes Ticket 1's target. The most important user journey from Day 2 becomes Ticket 2's target. The mysteriously-failing call from Day 3 becomes Ticket 3's target. The feedback-flywheel onboarding from Day 8 becomes Ticket 4's target. By end of Day 7, the first three tickets are in your tracker, scoped, owned, estimated, and ready for the next sprint.
Artifact → Four tickets. Scoped. Owned. Estimated.Evals onboarding day.
This is the day most teams skip and regret. Ground truth from Day 5 sitThe Observability & Evals cluster sits unbuilt.ls cluster sits unbuilt. The feedback flywheel has no place to run. Today wires the cluster up. Pick one trace-storage vendor: LangSmith, Braintrust, Arize Phoenix, or a homegrown equivalent. Pipe 100 real production traces in. Author the first five evals from Day 5's gap list. Run them. Record the baseline. Wire one into CI. “Quality > quantity; a small set of well-tagged evals covering the behaviors you care about beats thousands of noisy ones” (Vivek Trivedy, LangChain, April 2026).
You will know the eval suite is working when Andrew McNamara's bar holds for your team: “Once your judge reliably matches human evaluators, you can trust it at scale” (InfoWorld, April 2026). Until that bar lands, the LLM-as-judge is a rough signal you cross-check by hand. After it lands, it is a primitive that runs unattended. Sample 50 production cases; score them by hand and by judge, then compare. When the judge agrees with humans on more than 90 percent of cases, the suite is calibrated.
The resistance to this day is not a scheduling problem. It is a mental-model problem, and the industry data shows its exact shape. Nearly nine in ten teams have observability; barely half run evals (LangChain's 2026 survey of 1,300+ practitioners). That gap is the difference between watching the agent and learning from it. Harrison Chase's three-requirement test is the bar to hold Day 8 against: does your platform store traces, does it store feedback attached to those traces, and does it generate feedback from traces automatically? Until all three are yes, you have a logging system, not a learning system.
One primitive joined the kit while this episode was being revised, and it belongs on Day 8. TAn explicit, machine-checkable definition of what "done" looks like for a task, plus a forced continuation loop until the rubric is satisfied.rubric is satisfied. LangChain shipped first-class rubric support in June 2026; the pattern generalizes the goal-injection approach coding agents have used for a year to any workflow. Author one rubric per high-value user journey as part of Day 8. The rubric turns your eval suite from passive measurement into active steering. The evals tell you the agent failed; the rubric refuses to let it stop until it hasn't.
And if your agent runs long-horizon work, know what the 2026 implementation path looks like, because the eval unit itself changed shape this year. For multi-turn agents, a static input-output pair is too small a test. The failures that matter emerge across forty turns and three tool boundaries. The shipping answer is the eval as environment. Harbor, wired into theHarbors, wired into the LangChain stack, gives every trial a fresh containerized sandbox (LangSmith Sandboxes, Daytona, Docker, Modal, or E2B, pluggable). It runs hundreds in parallel with no shared state, and lands every run in observability as a dataset-plus-experiment with traces attached.ith traces attached. Day 8's stretch goal is one workflow, one containerized eval environment, and one failing production trace turned into a reproducible sandboxed regression test.
Artifact → 5 evals in CI. 1 baseline. 100 traces stored. 1 rubric per journey. Judge calibrated >90%.The open-harnRun one question across the five clusters: who owns the source of truth for each?the source of truth. Is it your team or a vendor? Identity. It includes system prompts, persona, and rules. Exportable? Is it in a repo you control? Memory Policy. It includes compaction, boot-time injection, and retrieval. Is it configurable, or vendor internals? Orchestration. It includes the main loop, Ralph Loop continuation, sub-agent handoffs, and routing. Can you swap models without rewriting? Interception. The six hooks (before_agent, before_model, wrap_model_call, wrap_tool_call, after_model, after_agent). Do you control them, or does the vendor? Observability & Evals. This includes the trace layer and eval suite. Are traces exportable? Are evals portable? Score each as portable, partial, or locked-in. A harness that scores "locked-in" on three or more clusters is not truly your harness. It's a rented harness wearing your product's name. This is fine for 60% of workflows. It is catastrophic for the other 40%.
June 2026 added two rows to this table, and both earned their place the hard way. Consider Consider Eval-infrastructure portability. Who stores your ground truth, rubrics, and regression history? OpenAI is suspended for all customers for nineteen days with zero notice (June 12 to July 1). Model-agnostic routing is business continuity, not cost optimization. A fallback you have never evaluated is a hope, not a fallback.
Also add one forward-looking row. This is for billing attribution. Can you attribute token cost per cluster, per workflow, per failure, before metered agentic billing arrives? Anthropic announced, then paused on June 15, the separation of programmatic agent usage into API-rate credit pools. The date moved. The direction did not.
Artifact → 5-row table plus eval-infra, model, billing portability.The trace is the ground truth. Dashboards summarize, but they also lie. Reading fifty traces end to end is the only way to see the agent the way your user does.
Harness Engineering · Episode 6Part 2, Where You Sit on the Harness Maturity Ladder
Five rungs. Your team is on one of them right now. Knowing which one is more useful than any roadmap.
Rung 1: "We Have a Prompt"
A string and a model call. No wrapping. Retry is exception handling. Logging is print() statements in a utility file. Nobody owns the prompt file, so everybody edits it.
Teams at Rung 1 are usually in their first ninety-days with the agent. Demos work, but production is a coin flip. The team believes the problem is the model.
Move to Rung 2 by adding retry logic and basic output parsing. This takes two days of work.
Rung 2: "We Have a Retry"
They have naive retries, maybe some JSON parsing. If the model returns malformed output, the harness retries with the same prompt and hopes. If validation fails after three attempts, the harness raises an exception.
Teams at Rung 2 have stopped crashing on every malformed output but haven't stopped drifting. Sessions recover occasionally. The team believes the problem is now half the model and half the plumbing.
This is where most teams are. Roughly sixty to seventy percent of enterprise teams I talk to sit here.
Move to Rung 3 by adding structured output enforcement, ground-truth evals, and drift detection. This takes one to two sprints.
Rung 3: "We Have an Eval Suite"
Ground truth exists. Structured outputs are enforced at the provider level. The team runs an eval suite before every model change. Drift detection is in place. The team knows in a day when the agent's behavior has shifted.
Teams at Rung 3 have left the dark. They can measure. They can compare. They can improve with evidence instead of hope.
The jump from Rung 2 to Rung 3 is the highest-ROI transition in a harness program. Most of the pain disappears here. The team believes the problem is nuanced now. It's partly the model, partly the patterns, partly the org.
Move to Rung 4 by adding the full eleven components from Episode 2. This takes one to two quarters.
Rung 4: "We Have a Harness"
All eleven components exist. Observability traces every call. Retries carry context forward. The tool registry is narrow-gated. A safety layer runs before every consequential action. The team has a name for their harness and a diagram.
Teams at Rung 4 ship new features without regressing reliability. The harness compounds. The team believes the harness is a product now, not plumbing.
Move to Rung 5 by establishing discipline around versioning, migrations, and a flywheel. This is less about code and more about org design.
Rung 5: "We Have a Harness Discipline"
Every harness component is versioned. Migrations are written when shapes change. Evals feed back into prompt templates. Prompt templates feed back into evals. There is a named harness owner. This person's job is to tend the harness, not to ship the next feature.
Teams at Rung 5 are rare. They are also the teams shipping agents that get renewed year after year, absorbed deeper into enterprise workflows, and raised in autonomy limits without rebuilding. The harness discipline is what turns an AI product into a compounding asset.
Most teams are at Rung 2 trying to act like Rung 4. The maturity ladder isn't a vanity metric. It's a diagnosis of which problems you are equipped to solve.
Harness Engineering · Episode 6Part 3, Four Tickets to Ship Before the Next Sprint
Four tickets. Each one is small enough for a single sprint, measurable by Friday of the sprint after it lands, and structured so nothing gets vague when it hits the backlog. Instantiate these templates against the audit artifacts from Part 1.
- Why
- This call runs [X] times per day. Every failure here cascades into [specific downstream system]. We currently have no output schema enforcement. Drift enters through this call first.
- Scope
-
- Define a JSON schema for the call's response, with
additionalProperties: false. - Replace every free-form string with a regex-bounded type or an enum.
- Turn on strict mode at the provider level.
- Add a bail-rate telemetry metric (how often the model emits empty versus incorrect responses).
- Roll it behind a feature flag. A/B test against current behavior.
- Define a JSON schema for the call's response, with
- Done When
- The schema is in production behind a flag. Parse-error metrics drop measurably. The flag becomes default-on the following sprint. The pattern is documented as the default for all new schemas this quarter.
- Why
- This journey has no automated eval today. We discover failures through user complaints or on-call escalations. That is a lagging signal, which costs us trust.
- Scope
-
- Select 30 to 50 representative sessions from the last quarter. Use a mix of successful, borderline, and failed sessions.
- Write the ground-truth label for each session: what the agent should have done, in structured form.
- Build an eval harness that replays each session against the current model and scores against ground truth.
- Run the eval against the current model. Record the baseline.
- Wire the eval into CI so every model or prompt change triggers it.
- Done When
- The eval runs on every merge to main. The baseline is documented. A failure threshold is set. Any regression above [X]% blocks merge. A dashboard shows the eval score over time.
- Why
- We cannot diagnose this call today. When it fails, we have the user's complaint and the model's final output. We do not have the prompt that was assembled, the context window's contents, the tool calls made, the retries attempted, or the time spent in each stage. The call is a black box.
- Scope
-
- Add a trace ID to every call.
- Log the fully assembled prompt, not just the template.
- Log every tool call, its arguments, and its response.
- Log every retry, including the error that triggered it and the augmented context that was sent.
- Pipe traces to whatever observability system the team already uses (Arize, LangSmith, homegrown. This doesn't matter; pipe to one.
- Done When
- Every production call emits a trace. On-call can diagnose a failed session in under ten minutes. One existing open incident is closed using the new tracing.
- Why
- Observability and evals sit in Episode 2's fifth cluster for a reason. They are the engine that makes every other cluster improve. Running them as separate primitives (traces in one place, evals in another, edits informed by neither) is how most teams run them today. The flywheel is what wires them into one loop.
- Scope
-
- Pick the failure mode with the highest incidence from Day 8's five baseline evals.
- Mine fifteen more trace examples of that failure from production. HanPropose one concrete harness edit, a prompt change, a schema tighten, a tool rename, or a middleware hook, that addresses the pattern.t addresses the pattern.
- Split the new eval set 70/30 into optimization and holdout. Validate the edit against the holdout befIf the failure mode involves ambiguous completion criteria, author a rubric that defines "done" in machine-checkable terms and wire it into the continuation logic. the continuation logic. This is now a first-class, portable primitive. This is not a custom orchestration hack.
- Document the cycle in a one-page retrospective. File it next to the eval suite.
- Done When
- One failure mode has gone from observed in traces, to encoded in eval, to addressed by a specific harness edit, to validated on holdout, to shipped. The one-page retro explains what changed and why. The flywheel has turned once. This cycle is the template for every future cycle.
Four tickets. This requires two days of scoping. This requires one sprint of work. By By By Friday of the next sprint, the dashboard moves. This is what Monday morning looks like.
Harness Engineering · Episode 6Two Stretch Tickets. For the Team That Finishes Early
When this call fails today, reconstruction means reading logs in sequence and guessing what the agent saw. Sandbox replay lets the on-call engineer re-run the exact agent state. This includes the same session, tool inputs, and context. All in an isolated environment. That cuts diagnosis time and hands you a reproducible failure case for eval authoring in the same motion. Managed runtimes increasingly provide this out of the box; isolated-execution vendors cover teams running their own. Done when: any production failure of this call can be replayed within ten minutes of the incident being filed.
The observation-without-feedback gap applies to your team too: the traces exist, the evals do not. Pull 500 sessions. Apply the failure taxonomy from Day 2. Find the three failure patterns no existing eval covers. Author one eval per pattern, wire them into CI, and write down the mining protocol so it runs quarterly.
Part 4, What to Tell Your CFO, Head of Eng, and CEO
There are three conversations, covering the same investment, with three different translations. You will have versions of each of these in the first two weeks after the audit lands. The words matter.
Why it lands. Engineering responds to this framing because it matches how they already think about systems. The harness is not a product concept to them. It is a set of components, most of which they wrote, that currently lack coherence. They will recognize the audit output as the artifact they wished existed.
Why it lands. Design responds to this framing because it connects the invisible infrastructure to the visible experience. They have been describing the problem in user language for months. The harness is the engineering answer to their design research.
Why it lands. Exec teams respond to this framing because it is the language of investment, not engineering. They are not evaluating whether a retry loop should be rewritten. They are evaluating whether to preserve future optionality. That's a decision they know how to make.
You should audit, diagnose, ship, and convince. One month.
The kit compresses into four weeks. Each week has a single artifact, a single outcome, and a single measure of done. Do it once, and your agent is more reliable. Do it four times, and your team has moved two rungs up the ladder.
-
Week 1
Audit.
Nine days. It produces nine artifacts. It also produces one diagnosis paragraph placing your team on the maturity ladder.
-
Week 2
Tickets.
There are four tickets scoped, owned, estimated, and in the backlog. Evals onboarding (Day 8) and the open-harness audit (Day 9) feed Ticket 4.
-
Week 3
Conversations.
Engineering alignment is key. Design alignment is key. Exec alignment is key. Use the three scripts, adapted to your team.
-
Week 4
Ship.
The first of the four tickets is in production behind a flag. The dashboard is moving. The baseline is recorded. If your eval work surfaced a journey with fuzzy completion criteria, author the rubric now, while the context is fresh. Verification primitives ship in days, not quarters.
Do it twice, and your agent is noticeably more reliable than it was a quarter ago. Do it four times, and your team has movedDo it continuously for a year, and you will have built a harness discipline: the thing that separates teams still shipping agents in 2027 from teams that quietly wound down their AI programs when reliability never arrived.liability never arrived.
The kit isn't clever. It's disciplined. The compounding isn't from any one sprint. It's from running this same cycle every month while everyone around you is still arguing about which model to pick.
Harness Engineering · Episode 6Enterprise Lens. Patience is the Enterprise Weapon
Startups have to hit PMF in ninety-days. Enterprises can run the feedback flywheel for ninety weeks, and the time horizon itself is a weapon. The mistake most enterprise PMs make is importing the startup cadence because it looks faster.
The cadence that actually compounds at enterprise scale is quarterly harness OKRs, not sprint ones.
The instructive arc is Bloomberg's trajectory with BloombergGPT. The May 2023 paper announced a 50-billion-parameter model trained on financial data. Independent and internal assessments over the next year showed that for several high-value financial tasks, off-the-shelf frontier models combined with retrieval and evaluation were producing competitive or better results at a fraction of the cost.
Rather than doubling down on the training bet, Bloomberg's applied AI teams pivoted toward RAG plus evaluation plus targeted harness investment, and compounded improvements across twelve-plus months of quiet iteration. The patience was the bet.
The PM move is to set quarterly harness improvement OKRs, not sprint ones. The nine-day kit is a sprint. The compounding is quarterly. Run the kit once, and you have moved one ticket. Run it four times in a year, and you have moved the maturity ladder from Rung 2 to Rung 4. Run it twelve times, one kit per month for a year, and you have built a harness discipline. That is Rung 5, and it is what separates enterprises still shipping agents in 2027 from enterprises that quietly wound down their AI programs when reliability never arrived.
Patience is not slowness. Patience is running the nine-day kit again, on the next workflow, while your competitor is still arguing about whether to buy or build.
You've got the kit. But the most ambitious teams aren't just adding a harness. They're reorganizing around it.
The kit works on any team, at any scale, against any agent. It works in a ten-person AI startup and it works in a Fortune 100 AI center of excellence. The tactical floor is the same everywhere. What changes is what comes next.
The most ambitious teams I've watched over the last year aren't just adding a harness. They are reorganizing around it. The harness stops being a layer under the product and starts being the product. The PM's role changes. The engineer's role changes. The operating model changes. Roles that didn't exist eighteen months ago now run half the roadmap.
That's Episode 7.
Sources & Further Reading
Anthropic: Effective Harnesses for Long-Running Agents (November 2025). Audit framing, maturity ladder foundation.
OpenAI Codex Best Practices (February 2026). Tracing and structured output as reliability primitives.
Vivek Trivedy, LangChain, “Better Harness: Hill-Climbing with Evals” (April 2026). Evals onboarding day, quality > quantity framing.
Sydney Runkle, LangChain. Rubric support shipped in deepagents (June 2026). Rubrics as a Day-8 primitive; eval suite as active steering.
Harrison Chase, LangChain. Agent Observability Needs Feedback to Power Learning (May 2026). The three-requirement test; “traces tell you what happened, feedback tells you what it meant.”
LangChain: State of Agent Engineering 2026. The observability-without-evals gap across 1,300+ practitioners.
Heath Ramsey (ServiceNow) and Andrew McNamara (Shopify), in InfoWorld (April 2026). Starting move and judge-calibration bar.
BloombergGPT press release and the May 2023 paper. Enterprise patience arc and the pivot toward harness + RAG + evals.
Practitioner interviews across enterprise AI programs, 2025–2026. Nine-day audit pattern abstracted from field observation.
Previous in this series: Episode 5. What It Costs, What It Returns
Key Takeaways
- Draw the map before you touch anything. Write down what the agent actually is: the parts, who owns each part, and where it tends to break. You can't fix a system you haven't labeled, and most teams haven't.
- Read real sessions with your own eyes. Dashboards summarize, and summaries hide the interesting failures. Reading enough real conversations end to end is the only way to see the pattern behind the noise. There is no shortcut.
- A few evals that catch real bugs beat many that catch imaginary ones. Your test suite becomes useful the moment each test is tied to a failure someone actually pays for. Coverage matters much less than aim.
- Measure how fast the agent recovers, not just how often it's right. Users forgive an agent that notices a mistake and fixes it. They don't forgive one that keeps going as if nothing happened. Recovery is the number that best predicts trust.
- Small loops beat big sprints. These programs are won by running the same simple cycle: look, diagnose, fix, measure, many times. The cadence is the deliverable; the tickets are just what happens inside it.
- Patience is the one advantage a fast competitor can't copy. Anyone can win a quarter. Almost no one wins twelve in a row. If the loop is working, keep running it, time itself is the moat.
How this episode connects to the rest of the stack
Each link points to the same problem, told from a different seat.
- Agentic · T10. Your First Context PipelineThe architecture-side equivalent of a Monday kit.
- Agentic · T19. Evaluating the HarnessWhat to measure once the kit is in place.
- Evals · T10. Your First Eval SuiteThe eval-side Monday kit.
- AI PM OS. The Operating SystemHow the kit slots into the broader PM workflow.