Prove the Environment Is Working

Vocabulary for this episode

Coined here: Environment Operating Contract (the artifact). Three claims of health (execution, control integrity, decision quality — they fail independently). The four quiet failures, including compensatory degradation (the agent adapting around a plausible but weaker dependency answer). Six evidence classes. Instrument drift (the measurement changed, not the system). Trajectory invariants as an environment-conformance test. Durable resolution as a unit-economics denominator.

What you own

This is where the PM's role stops being advisory.

Your team owns how it is measured. You own what must be known, and what decision follows.

Skip this division and one of two things happens: you become a junior data engineer with a PM title, or you accept a dashboard someone else designed and lose the ability to tell whether your product is working.

Where this sits in the body of work

This episode deliberately does not teach evaluation. The AI Evals & Observability series builds that discipline. Environment Engineering does something upstream.

ENVIRONMENT ENGINEERING (Ep 01-04)
Declares what must be true around the agent
  - Consequence is bounded
  - The world is as declared
  - Authority is attributable
  - Compromise is contained
                |
EPISODE 05 - THIS ONE
Converts each declaration into a testable operating
claim with an owner, an evidence standard, and a response
                |
AI EVALS & OBSERVABILITY
Builds the machinery that tests those claims
                |
THE AI PM DECISION
Ship - hold - degrade - roll back - expand autonomy

The Environment series states what must hold. The Evals series proves whether it does. Episode 05 is the contract between them.

Read The Quality Owner alongside this — it establishes why the PM defines "good." Read Production Watch for why release gates and production monitoring solve different problems.

01Two promises, and why teams confuse them

Every production agent makes two promises. Most teams instrument one.

The system promise. The environment will provide the declared tools, data, identity, limits, and degraded behavior.

The product promise. Operating inside that environment, the agent will do work that is useful, correct enough, and economically defensible.

Episodes 01–04 built the first. Keeping it does not keep the second.

Completion proves termination, not correctness.

Every row on the right is a product question. Every row on the left is something a platform team can produce without you. Your first job is to stop accepting left-column evidence for right-column claims.

02Three claims of health

An environment is healthy only when three separate claims can be defended. They fail independently, have different owners, and no single score represents all three.

Hold them apart, because they move in opposite directions and still average to "fine." A system can complete every run while making progressively worse decisions. It can make excellent decisions while a permission boundary silently widened. It can refuse so much that control integrity looks perfect and the product becomes useless.

Reliability, control integrity, and decision quality are separate product claims. A green average can hide a red one.

Your operating rule: when a review presents one combined "agent health" number, ask which of the three failures has been averaged away. Asked consistently, that question is worth more than any dashboard you could commission.

Add saturation to the reliability column

Most agent reliability reporting covers latency, throughput, and errors and stops. The SRE golden signals are latency, traffic, errors, and saturation — how full the system is relative to capacity.

Saturation matters disproportionately for agents, because the system can preserve completion by degrading the work. Context windows near their limit force truncation. Tool queues back up and trigger fallbacks. Rate limits push the agent onto a cheaper model. The human review queue saturates, and the control you depend on becomes unavailable.

That last one is a product signal, not a platform signal. If your high-consequence approval queue is forty hours deep, your approval control has stopped existing, regardless of what the policy says.

03Turn Episode 01–04 promises into evidence

The mechanical core, and the deliverable you personally produce.

If an environment promise produces no observable evidence, it is not an operating promise. It is a claim you are hoping is true.

Episode 01 — consequence is bounded

Episode 02 — the declared world existed

Episode 03 — authority was attributable

Episode 04 — compromise was contained

Produce this mapping before you produce a dashboard. Two hours, and it will change what your team builds.

04Start from decisions, not metrics

The most common way an AI PM loses control of observability is starting with "what can the platform give us?" That produces a dashboard of available metrics, not a decision system.

Invert it. For every operating claim: what decision does this inform, who has authority to make it, how fast, what action follows, and how much uncertainty can that action tolerate?

A signal with no attached decision is a number. A signal with a decision, an owner, and a response is a control.

Anything failing this test belongs in a diagnostic view, not an operating review.

05The four quiet failures

Security failures announce themselves. Product failures do not.

Judgment drift. The decision distribution shifts while every individual decision remains defensible and authorized. Approval rate moves from 34% to 61% over five weeks. No single approval is wrong. The aggregate is a different policy than the one your domain owner wrote. This was the opening incident. Inspecting one run at a time finds nothing, which is precisely why it survives six weeks.

Input drift. The population changes and the agent does not: new merchant categories, longer documents, a new fraud pattern, more ambiguity. The agent has not degraded. Its fitness has.

Substrate drift. Model version, prompt template, policy bundle, tool schema, retrieval corpus, or evaluator version changes underneath a stable product surface. Every quality signal must be attributable to the versioned substrate that produced it. Pinning a model version is not sufficient.

Compensatory degradation. The most dangerous, and the one your reliability dashboard is structurally incapable of catching. A dependency returns a plausible but weaker answer and the agent adapts without failing.

The customer-history endpoint still returns HTTP 200. Replication lag omits recent disputes. Tool success rate stays at 100%. The agent receives incomplete history, treats repeat disputants as first-time claimants, and approves more. Nothing errored. The environment simply has less truth in it.

The most dangerous dependency failure is not an error. It is a plausible answer with less truth in it.

Dependency health cannot be measured by availability alone. It must include the semantic properties your decisions depend on: freshness, completeness, provenance, consistency. That requirement comes from you, because only you know which properties the decision rests on.

06Drift is change, not harm

This correction matters more than it first appears, and getting it wrong costs organizational credibility. A distribution can move because the population improved, a policy changed intentionally, cases got easier, an intervention worked, a season shifted — or because the system regressed.

Drift is evidence of change, not evidence of harm.

The last row is the one teams miss. A judge-model upgrade changes the instrument, not the system. React to instrument drift as regression and you will roll back a healthy version.

Attribution requires more than correlation

Saying a prompt change "shifted the agent's posture" is a story, not a finding.

A correlated version is a suspect. A replay or a concurrent control makes it evidence.

Enforce this personally. The pressure in an incident room always runs toward the first plausible story.

07The substrate has an expiry date

§05 treats substrate drift as something that happens to you. A large part of it is scheduled, announced in advance, and ignored until it becomes urgent.

Model providers publish deprecation notice periods. OpenAI commits to at least six months for generally available models, at least three months for specialized variants, and as little as two weeks for preview models — while explicitly recommending against preview models for business-critical workloads unless you can migrate on short notice . Google's model deprecation works the same way: a deprecation date starts a window during which the endpoint keeps working for existing workloads but receives no updates, followed by a retirement date . Internal API decommissioning conventions are typically shorter, at 30 to 90 days .

Read those numbers as a planning constraint. Six months sounds generous until you count what a model migration actually requires under this contract.

Why migration is not a swap

The last row is the binding constraint. If your durable-resolution window is 60 days, you cannot demonstrate the successor is equivalent in less than 60 days of production traffic — regardless of how good the offline replay looks.

A six-month deprecation notice is not six months of runway. It is six months minus your evidence-accumulation window, minus your migration engineering, and the evidence window cannot be compressed by adding people.

What the contract must record

The last row is the honest one. Sometimes the deadline wins. The decision then is not whether to migrate but which gate to re-tighten while the evidence catches up — narrower autonomy, higher review coverage, or a lower consequence ceiling until the successor has earned its baseline back.

What you own: knowing what is on a preview tier, and refusing to build a release gate on an instrument that can be retired with two weeks' notice.

The question: "Which of our substrate dependencies has the shortest notice period, and what is our effective runway on it?" If nobody knows, you have an unscheduled migration in your future.

08Specify trajectory invariants, not one correct path

The Agent Evals chapter covers trajectory evaluation. Episode 05 uses it narrowly: verifying that the environment's required operating sequence actually occurred.

The trap is demanding one canonical path. Adaptive agents have multiple legitimate routes, and forcing imitation destroys the value you deployed an agent to get. Specify invariants — properties holding across every valid path.

Evaluate trajectories by invariants, not by imitation of one preferred path.

Every invariant traces to an earlier episode: provenance from 04, authority sequence from 03, idempotency from 01. Trajectory invariants are how Episodes 01–04 become testable.

One constraint on what you can observe

Do not build invariants on the model's stated reasoning. Generated rationale is output, not a faithful account of internal computation, and storing it carries privacy and security cost. Build on observable execution: tool calls, arguments, results, state transitions, retrieved sources, policy decisions, authoritative outcomes.

09Not all evidence is equal

Your most common failure in an operating review will be accepting weak evidence for a strong claim.

Correctness is not one label. It is an evidence claim assembled from facts, policy judgments, and consequences.

Never promote a class. A stable approval rate is not proof of correctness. A reviewer label is not proof the customer outcome was good. A low appeal rate is not proof of satisfaction, because only some people appeal. A judge score is not a fact about the world.

The timing problem you cannot engineer away

The evidence you trust most arrives too late to prevent the loss. The evidence that arrives fastest is the evidence you trust least.

A behavioral signal is available in minutes and proves almost nothing. A chargeback is nearly conclusive and arrives in sixty days. The resolution is not to pick one: use fast internal signals to detect and slow external outcomes to calibrate, treating every confirmed downstream failure as a test of whether your leading indicators flagged it.

The maturity of your quality program is measured by how reliably its fast signals predicted its slow ones.

10Evaluators are instruments, including the human ones

The evals series develops judge validation and human review in depth. Smarter Judges for calibration, Human-in-the-Loop for sampling and routing. Episode 05 needs the operating rule those chapters produce, applied evenly.

Every evaluator — model or human — is an instrument. Trust begins when its error is measured.

Teams apply appropriate skepticism to LLM judges and none to human reviewers. For ambiguous policy decisions that asymmetry is not defensible.

For machine judges. The literature documents position bias, verbosity or length bias, and self-enhancement bias, along with mitigations. Treat those as known instrument faults to test for, not reasons to abandon judges. Calibrate against domain-reviewed cases from your population, not a generic benchmark. Test for position, verbosity, self-preference, and same-family effects. Recalibrate on every judge-version or rubric change. Use scores for triage and population trends, not per-case verdicts. Route disagreement to human review.

For human reviewers. Use a versioned policy rubric, not general judgment. Dual-label a calibration sample and measure agreement. Adjudicate disagreements and track the overturn rate. Track agreement by policy clause — low agreement localizes an ambiguous rule. Keep the original approver out of the post-action audit for the same case. Periodically relabel a stable anchor set to detect reviewer drift. Mature ML operating guidance says the same: sample randomly, oversample high-risk cases, route ambiguous ones to experts, measure agreement.

What you own is not the statistics. It is the acceptance criterion — what evaluator error rate is tolerable before that evaluator's output becomes a release gate. Task-specific. Do not import a number from a vendor blog.

Reviewer capacity is a control, not a resource

§02 flagged review queue depth as saturation. The stronger statement: human review is a control in Episodes 03 and 04, and controls have availability.

Episode 03 made approval a boundary for exception refunds. Episode 04 qualified it — a boundary only when reviewers are not flooded into rubber-stamping. This episode has to measure that.

The rows that matter most are approval rate by reviewer and time spent per case. A reviewer approving 99% of cases in eleven seconds is not a control. They are a latency cost with a compliance signature attached, and the audit trail will say a human approved it.

A control that is always saturated is not a slow control. It is an absent control that still appears in the design document.

The response when capacity is exceeded is not to hire faster. It is to reduce what requires review — narrow the eligible population, raise the threshold that triggers escalation, or lower the autonomy ceiling until capacity recovers. All three are product decisions.

Selective evidence

Feedback is not evenly distributed. Approved refunds generate outcomes. Denied claims often never reveal whether approval would have been better. Escalated cases get more scrutiny. Complaints come only from customers who complain.

A random audit gives an unbiased estimate of reviewer-labeled policy conformity across the sampled population. It does not manufacture counterfactuals for actions the system did not take. This is the selective labels problem, named in the ML literature: when outcomes are observed only for cases a decision-maker acted on, the unobserved cases cannot be scored after the fact.

State this limit out loud in reviews. It protects you from claims your evidence cannot support.

11Your evidence system is a second production system

This episode asks the team to record prompts, tool results, customer messages, rationales, authoritative parameters, and full traces. That creates a second store of highly sensitive data — and it belongs in Environment Engineering, because the evidence system is part of the world around the agent.

OpenTelemetry's GenAI semantic conventions give a vendor-neutral vocabulary for agent and tool spans, but content capture is opt-in and requires governance. Plan for churn: the conventions live in a dedicated repository and remained at Development status through 2026 with no committed stabilization timeline, while HTTP and database conventions have long been stable.

One boundary worth stating plainly for anyone reading a vendor pitch: OpenTelemetry standardises how a run is recorded. It does not score whether the decision inside that run was any good.

The read-access row is where Episodes 04 and 05 collide productively. Episode 04 needs read logging to scope a 72-hour disclosure assessment. Data minimization pushes the other way. Resolve it deliberately in this contract, not by whichever team configured the pipeline last.

Observability that reproduces the production data boundary has created a second production system. Govern it accordingly.

More visibility is not automatically better. You are trading evidentiary value against data exposure, and that trade is a product decision.

12Read the metrics like a PM

Every operational metric has a direction that feels good and a reading that is wrong. Name both before it goes on a dashboard.

Completion rose. Fewer crashes — or the agent is resolving cases it should escalate. Did completion rise because the product improved, or because judgment got less conservative?

Refusals rose. An attack, input drift, or boundaries correctly rejecting what they were built to reject. A falling refusal rate deserves more scrutiny than a rising one. Which refusal reasons changed, in which segments, after which version?

Escalations fell 40%. Genuine capability growth — or decisions beyond the agent's competence. Did escalation fall while independently reviewed escalation appropriateness held steady?

Latency fell. Efficiency — or skipped verification steps. Which trajectory invariants changed alongside it?

Cost per run fell. Efficiency — or less evidence retrieval and weaker deliberation. What happened to fully loaded cost per durable resolution?

A durable resolution conforms to policy, reconciles correctly, requires no reversal, produces no repeat contact inside the observation window, and carries its share of model, tool, review, rework, and incident cost. Cost per run falling while cost per durable resolution rises means the agent got cheaper at being wrong.

The AI PM's job is not to celebrate metric movement. It is to identify the alternative explanation.

13Quality objectives under uncertainty

SRE gives the mechanism: an SLO sets a target, the error budget defines acceptable shortfall, and the remaining budget gates releases. The budget is also why a freeze is a product decision rather than an engineering one — it makes the trade between shipping speed and accepted risk explicit.

Agent quality needs one adaptation. Correctness is sampled, delayed, and consequence-weighted, so the budget is an estimate with error bars.

A quality budget estimated from a sample is a distribution, not a balance.

Do not present the estimate as exact. Equally, do not let uncertainty become the reason you have no gate.

What "freeze" actually means

The naive version — all changes freeze — is unworkable and will be ignored the first time it fires. When the budget is exhausted: freeze autonomy expansion and non-remedial prompt, model, and policy changes; permit controlled rollback; permit safety and remediation changes; increase review coverage where the failure concentrated; require re-established evidence before expanding; handle new versions as cohorts.

A related trap: exempting a new version's output from quality objectives while it is "being re-baselined" hides regressions during the window of maximum uncertainty. Every substrate change starts as a distinct cohort. Its outputs count toward total exposure immediately but are not pooled with the incumbent baseline. Promotion requires a predeclared comparison against the incumbent with no safety or quality breach.

14Bound the fleet, not just the run

Every control in Episodes 01–04 bounds a single run. Product exposure is a fleet property. A 2% error rate sounds tolerable. At 40,000 monthly decisions it is 800 wrong outcomes. Whether that is a rounding error or a regulatory event depends on which cases failed and how severe they were.

Every operating review needs: per-run rate; monthly affected volume; consequence-weighted exposure; version concentration; cohort concentration; trend and evidence delay.

A tolerable per-run error rate can still produce an intolerable monthly exposure.

A stable global average routinely hides one failing cohort, one bad prompt version, or a small high-consequence category driving most of the loss. Aggregate reporting without segmentation is how concentrated harm survives quarterly review.

15Close the loop with the people affected

Everything above is the system evaluating itself. Closed loops drift together.

The NIST AI Risk Management Framework is explicit that post-deployment monitoring plans should include mechanisms for capturing and evaluating input from users and other AI actors, appeal and override, decommissioning, incident response, recovery, and change management. The MANAGE playbook sets out each in implementable detail. Read it as monitoring design guidance, not a certification checklist.

Appeal and override are the parts teams skip, and they are the cheapest external evidence available.

A low appeal rate does not prove quality. A high override rate does not prove the agent is wrong. Read them as external anchors, not verdicts.

16Someone may require this by law

Everything above is framed as good practice. For a growing set of deployments it is a legal obligation, and the obligation shapes the design.

EU AI Act Article 72 requires providers of high-risk AI systems to establish and document a post-market monitoring system proportionate to the technology and its risks, actively and systematically collecting, documenting, and analysing relevant data — which may come from deployers or other sources — on performance throughout the system's lifetime, sufficient to evaluate continuous compliance . Where relevant it must include analysis of interaction with other AI systems .

Two structural details matter for how you build this.

The plan is part of the technical documentation, not an operational afterthought — it belongs in Annex IV documentation . The Commission was to adopt an implementing act laying down a template and the list of required elements .

Existing regimes can absorb it. Where a post-market monitoring system already exists under other Union harmonisation legislation, providers may integrate the Article 72 elements into it, provided equivalent protection is achieved — and this explicitly extends to financial institutions already subject to internal governance requirements under Union financial services law .

That second point is directly relevant to a billing dispute agent. If your organization already runs model risk management or operational risk monitoring, the correct move is integration, not a parallel system. Two monitoring systems disagreeing in an audit is worse than one.

What "continuous compliance" changes

The phrase does real work. It is not "was it compliant at conformity assessment." It is a standing obligation across the lifetime of the system.

What you own: determining whether your system is in scope, and if so ensuring the operating contract is written to be the monitoring plan rather than something that has to be translated into one later. Retrofitting documentation to a monitoring system built without it is substantially more expensive than designing them together.

The question: "If a regulator asked for our post-market monitoring plan tomorrow, would we hand them this document, or would we write a new one?" If the answer is a new one, the contract is not yet doing its job.

17The first ninety days

The contract in §18 describes a mature state. Teams reading this at launch need a sequence, because attempting all of it at once produces a dashboard nobody trusts and a review nobody attends.

Days 1–14 · Attribution before measurement. Nothing else works without it. Every run must carry a correlation ID, model version, prompt template hash, policy bundle version, manifest version, and cohort label. If you can only do one thing in the first two weeks, do this. Retrofitting attribution is expensive; its absence makes every later signal ambiguous.

Days 15–30 · Control evidence. Structural denials by reason code, egress blocks, provenance violations, invariant conformance. This comes second because it is largely mechanical, it is Episode 04's detection layer, and it produces value immediately without requiring a labeled dataset.

Days 31–60 · Decision distribution and the golden set. Segmented decision distribution, plus a fixed labeled case set. The golden set is the reference point everything later compares against. Build it before you need it — a golden set assembled during an incident is contaminated by the incident.

Days 61–90 · Human review and the first budget estimate. Calibrated rubric, dual-labeled sample, measured agreement, first quality budget with stated error bars. This is last because it is the most expensive and depends on everything above.

After 90 days. Judge calibration against the human-labeled set. Downstream outcome joins. Fully loaded cost per durable resolution.

Attribution first, control evidence second, distribution third, human labels fourth. Every other order produces evidence you cannot interpret.

18Translating this to your product

Five questions, whatever you build: Which of the three health claims does this dashboard cover? What is your durable-resolution denominator? Which instrument has never had its error measured? What is your effective runway on the shortest-notice substrate? Would a regulator accept this document as your monitoring plan?

19The artifact

20Anti-patterns

• "We have full observability." You have traces. Which product claim do they prove?
• "The dashboard is green." Which of the three claims?
• "Completion is 94%." Ninety-four percent of runs terminated. That is the entire statement.
• "Error rate is under 0.3%." Silent degradation raises no errors. That is its definition.
• "Escalations fell 40%." Against what change in escalation appropriateness?
• "No boundary was violated." A fully authorized action can still be wrong.
• "The human approved it." On authoritative data, in how many seconds, and who audited it afterward?
• "The judge agrees." With whom, on which calibrated population, under which rubric version?
• "Drift crossed the threshold." Drift establishes change. What establishes harm?
• "Quality looks flat." Segmented by version and cohort, or in aggregate?
• "The error rate is only 2%." State that as monthly affected customers and consequence-weighted exposure.
• "Cost per run is down." And cost per durable resolution?
• "We pinned the model version." Until the provider retires it. What is the runway?
• "We'll re-baseline the new model after launch." That hides regressions during maximum uncertainty.
• "Compliance will handle the monitoring plan." Then you will build the system twice.
• "We will add evals after launch." Then launch has no defensible definition of working.

21What product owns

1. Which promises must produce evidence — the Episode 01–04 mapping in §03.
2. What decision each signal informs, with an owner and a response.
3. Which semantic dependency properties the decision rests on.
4. Trajectory invariants — what must hold across every valid path.
5. Acceptance criteria for instruments — machine and human.
6. Reviewer capacity as a control, and what happens when it saturates.
7. The quality budget definition and what freeze actually means.
8. Segmentation — no quality decision on an unsegmented aggregate.
9. Evidence governance — what is captured, retained, and why.
10. Substrate runway — notice periods, evidence windows, forced-migration behavior.
11. Regulatory alignment — whether this contract is the monitoring plan.

The Environment artifact states what must hold. The eval system proves whether it does. The contract between them is yours.

The line

Completion proves termination, not correctness.

Reliability, control integrity, and decision quality are separate claims. A green average can hide a red one.

The most dangerous dependency failure is not an error. It is a plausible answer with less truth in it.

Drift is evidence of change, not evidence of harm.

The evidence you trust most arrives too late. The evidence that arrives fastest is the evidence you trust least.

A control that is always saturated is not a slow control. It is an absent one.

The AI PM does not own every signal. The AI PM owns the argument those signals must support.

22The AI PM page

Print this. Bring it to the room.

Decisions you own

Ten questions

1. Which of the three health claims does this dashboard actually cover?
2. What alternative explanation produces the same signal?
3. Is this evidence being promoted above its class?
4. Which version and cohort carry the change?
5. What is the monthly consequence-weighted exposure?
6. Which instrument has never had its error measured?
7. How deep is the review queue, and what is the per-reviewer approval rate?
8. What is your durable-resolution denominator?
9. What is the effective runway on the shortest-notice substrate?
10. Would a regulator accept this document as the monitoring plan?

Red flags

• One combined "agent health" score.
• Quality reported in aggregate, never segmented.
• A judge in a release gate with no measured error rate.
• Nobody knows the review queue depth.
• The golden set was built during an incident.
• A preview-tier model in production.
• The monitoring plan and the operating contract are different documents.

What you leave with

An Episode 01–04 evidence mapping. A decision table with owners. A trajectory invariant list. A substrate register with effective runways. An instrument acceptance criterion. A quality budget with stated error bars. A declared-limits section you are willing to read aloud.

One line to carry out: Completion proves termination, not correctness.

→Continue in AI Evals

Episode 05 defines the operating claims. The AI Evals & Observability series builds the machinery.

• The Quality Owner — why the PM defines "good."
• Traces & Observability — how execution becomes inspectable.
• Agent Evals — scoring outcomes and trajectories.
• Production Watch — release gates versus production monitoring.
• Human-in-the-Loop — sampling, routing, returning review.
• Beyond Pass/Fail — where single-score reporting stops being honest.
• Root Cause Analysis — from a moved metric to an attributed cause.
• Eval Economics — what the evidence costs, and what it is worth.

→Next · Decide What Runs Without You

Five artifacts now govern the environment. The Consequence Envelope bounds what one run can cost. The Environment Manifest defines the world. The Authority Matrix establishes authority. The Containment Model states what survives compromise. The Environment Operating Contract establishes how you know all four still hold.

Only now can the autonomy conversation happen honestly.

Not because the model seems capable. Not because completion improved. Not because human review is expensive and the queue is backing up.

Autonomy expands when a defined category of work has accumulated enough evidence — across world fidelity, control integrity, decision quality, and consequence — to justify removing one specific gate, with a predefined condition that puts it back.

That is not a platform decision or a security decision. It is yours. For the interrupt, freeze, and kill-switch mechanics underneath it, read Agent Governance; for why the evaluation is itself the requirement, read Evals as Strategy; and for the honest ceiling on all of it — evidence, not proof — read The Limits.