Your Monday Morning Harness Kit

QuestionWhat system is production running today?

Most teams already have a harness. They do not call it one.

The prompt lives in one repository. Retry logic sits in a utility. Tool permissions come from an API gateway. Traces land in an observability platform. A spreadsheet holds the eval cases. Three people own fragments and nobody owns the behavior produced by the whole.

Draw one page with four columns — Model, Harness, Tools, Environment — and add Runtime beneath them.

Inside Harness, add five rows: Identity, Memory Policy, Orchestration, Interception, Observability and Evals.

For each row, record:

Write “missing” when no implementation or owner exists. Do not fix anything today.

QuestionHow does the workflow fail in real use?

Read sessions, not final responses.

A session includes the request, context, tool calls, retries, state changes, and final outcome. The final answer can look correct while the path contains wasted loops, unsafe calls, or unverified effects.

Tag the six failure shapes from Episode 1:

For each failed session, mark the intended business outcome, the observed outcome, the earliest divergence, the layer that owned the preventable condition, and the evidence available.

Do not begin with an existing taxonomy beyond the six broad shapes. Read several traces first. New failure patterns may not fit your current labels.

QuestionWhat does the organization quietly work around?

Speak separately with the staff engineer and the person who takes the page. Ask two questions:

1. What do you repair manually without filing anymore?
2. If feature work stopped for one sprint, what would you fix first?

The first question surfaces hidden operating cost. The second surfaces accumulated judgment.

Listen for manual restarts, hand-corrected state, secret spreadsheets, prompt edits applied during incidents, repeated tool overrides, cases reviewers always reverse, and alerts everyone ignores.

The overlap between both interviews is often the first high-confidence ticket.

QuestionWhich tools create useful action, and which create confusion or risk?

Pull the current tool registry. Score each tool on four dimensions.

Then make one decision per tool: keep visible, rename, tighten the input contract, hide behind a skill, require approval, merge, or remove.

Do not delete the bottom third automatically. A rare escalation tool may be essential. A frequently used general tool may be the source of repeated errors.

Add two ownership fields: which identity calls the tool, and which team owns its permission and verification policy.

Detailed authority, reversibility, and protocol design belongs in The Tool Is the Contract. This audit needs the decision, not a protocol migration project.

QuestionWhich promised behavior can break without any test failing?

List the behaviors the product claims to support. For invoice reconciliation:

• MatchFull payments correctly
• PartialHandle partial payments
• DuplicateFlag duplicates
• ScopeStay inside tenant scope
• ExceptionAssign every exception
• QueueProcess the full eligible queue

For each behavior, ask:

If this broke tomorrow, which automated check would fail?

A blank answer is eval debt. Put one class at the top when it exists: we have no machine-checkable definition of done for the workflow.

Without that definition, every other eval measures pieces while the agent can still stop with the job unfinished.

Evaluate at the right altitude

Current agent-evaluation practice separates three levels:

A final-answer eval can miss a bad path. A run-level check can pass while the whole conversation fails.

QuestionWhat problem is the team equipped to solve next?

Use five rungs.

Do not average the whole organization into one score. Place the selected workflow. Write one paragraph:

We are at Rung [N] because [evidence]. The next rung requires [capability]. The smallest move is [ticket]. We will know it worked when [metric].

The reconciliation workflow is at Rung 2. We trace model calls but have no ground-truth test for full-queue completion. The next rung requires a workflow eval in CI. The smallest move is Ticket 2. We will know it worked when a pagination regression fails before release.

QuestionWhich changes are small enough to own and specific enough to measure?

The audit should compress into four tickets:

1. Enforce a structured contract on the highest-volume model output.
2. Add a ground-truth eval for the highest-consequence journey.
3. Trace the call the on-call team cannot explain.
4. Complete one trace-to-eval-to-change cycle.

Each ticket needs a named owner, one workflow, a baseline, acceptance criteria, a counter-metric, and a rollout and rollback plan.

QuestionCan the team turn production evidence into a release decision?

LangChain's 2026 survey found that 89% of respondents had some agent observability, while 52.4% ran offline evaluations on test sets and 37.3% ran online evaluations. Many teams can watch a run without converting what they see into a test.

Start smaller than your ambition.

1. Manually review 20 to 50 real traces with a domain expert.
2. Define success for one task.
3. Write five eval cases from Day 5's gaps.
4. Run them against the current system and record the baseline.
5. Put one regression eval in CI.
6. Attach feedback to the trace or thread it judges.

A trace tells you what happened. Feedback records what that behavior meant.

Calibrate judgment without a magic threshold

Some properties can be checked with code: required field, date range, source count, permission.

Semantic properties may need a model judge or human: whether the response resolved the user's intent, followed brand tone, or chose a reasonable path.

Calibrate model judges on labeled examples from this workflow. Prefer clear binary or categorical decisions. Review disagreements. Keep human judgment on consequential cases. Recalibrate when the agent, rubric, model judge, or traffic changes.

There is no universal sample size or agreement percentage that makes a judge safe for every workflow.

QuestionWhich product judgment remains portable if a vendor changes or disappears?

Run the question across five clusters.

Score each as owned and portable, configurable but vendor-dependent, or locked inside the product.

Add two continuity rows: has the fallback model passed the same workflow eval, and can the runtime export enough state to resume or migrate unfinished work?

A closed system may be correct for a commodity workflow. The audit prevents the decision from happening by accident.

Environment containment and reach need a separate operate-side audit. The World Around the Agent provides that method. Day 9 only records who owns the boundary and where the evidence lives.