Build Safe Realism

Three terms

Coined here: the Fidelity Plan, evidence ceiling, fidelity debt, and reachability classes RC-0 through RC-4 are operating framings for this series, not industry standards.

What you own

This is the episode where the ownership question is most often answered wrongly, because the environment in question does not look like a product.

The last row is where the July 2026 lesson actually lands. Test environments are routinely excluded from the review process that governs production, on the reasonable-sounding grounds that they are not production. That exclusion is a product decision by default, made by nobody, and it is the decision that failed.

An environment that can reach a real system is a production environment. What the organization calls it is not a control.

Where this sits in the series

EPISODE 01  Consequence Envelope   what one run may cause
EPISODE 02  Environment Manifest   the world it reasons in
EPISODE 03  Authority Matrix       whose authority it carries
EPISODE 04  Containment Model      what survives compromise
EPISODE 05  Operating Contract     how we know it still holds
EPISODE 06  Autonomy Ladder        what the evidence has earned
        |
EPISODE 07 — THIS ONE              Fidelity Plan
Governs the environments where that
evidence is produced, using the same
five artifacts turned inward
        |
EPISODE 08  Readiness Gate         whether the whole thing ships

Episode 06 made this episode necessary rather than optional. Its seven promotion conditions demand production-grade evidence: reversal paths exercised, demotion triggers fired in drills, seeded cases replayed, shadow comparisons run at volume. Each of those requires an environment realistic enough to produce a meaningful result.

The uncomfortable consequence: an environment realistic enough to generate promotion-grade evidence is realistic enough to cause a real incident. That is the fidelity paradox, and the rest of this episode is about resolving it dimension by dimension rather than with a single dial.

01The fidelity paradox

State the problem precisely.

The properties that make an environment evidentially useful can also make it operationally dangerous. The mistake is importing both as one bundle.

A low-fidelity environment can produce clean results that fail to predict production. A high-exposure environment can produce valid evidence while creating consequences the claim never required. The solution is not to choose a midpoint on one realism dial. It is to increase fidelity deliberately while constraining exposure independently.

Fidelity dimensions

Exposure dimensions

This separation changes the review. The question is no longer how realistic is staging? It is which production property must be reproduced for this claim, and which exposure can be removed without weakening it?

A Fidelity Plan deliberately increases fidelity while independently constraining exposure.

Temporal fidelity is often inexpensive and neglected. Consequence exposure is expensive and frequently acquired by accident. The July 2026 evaluation needed enough interface and network fidelity to test cyber capability. It did not need an uncontrolled route to a third party's production systems12.

02Classify by reachability, never by label

The prologue stated this rule in one line. This episode operationalizes it.

The words production, staging, UAT, sandbox, dev, and eval describe intent. They do not describe capability. An environment's actual risk is determined by what it can reach, what authority it holds, and whether its actions take effect — regardless of which of those six words appears in its name.

Replace the label with a class.

Two rules follow, and both are worth defending in review.

An environment inherits the highest reachability class of any path available from inside it. An otherwise-sealed environment with one allowlisted internal service is RC-3, not RC-0 with an exception. The July 2026 environment was described as highly isolated with limited access to one internal service1. Under this classification it was RC-3 from the moment that access existed, and everything downstream followed from it.

Exceptions do not sit outside the classification; they determine it. Governance attaches to the class, not the name. An RC-3 environment gets an Environment Manifest, an Authority Matrix, and a Containment Model, whether it is called production or called an eval harness.

The escalation chain, applied to test environments

Episode 04 established a six-link causal chain from code execution to organizational consequence. July 2026 walked it end to end, and it walked it inside a test environment.

Read the right-hand column as a design specification. An RC-1 environment breaks the chain at link two and cannot proceed. An RC-3 environment with credentials outside the execution boundary — the prologue's principle — breaks it at link three. Neither requires the sandbox to be perfect, which is the entire argument for treating test environments as designed systems.

03Data fidelity, and the two words that are not synonyms

Data is the dimension teams reach for first, because it is the most obviously relevant to decision quality. It is also the dimension with the clearest legal constraint and the most commonly confused vocabulary.

The distinction is not stylistic. It determines whether a regulation applies at all.

The test is a single question: can the transformation be reversed with a key? If yes it is pseudonymisation and GDPR continues to apply; if not it is anonymisation and GDPR does not5. Article 4(5) defines pseudonymisation as processing such that data can no longer be attributed to a subject without additional information kept separately, while Recital 26 exempts data where identification is not possible using all means reasonably likely to be used5. Article 32 lists pseudonymisation among the technical measures organizations must consider5.

The operational rule is blunt: test environments must never contain unmasked personal data from real users, and what you cannot use is a raw production export, a live production database connection, or any dataset where individuals could be re-identified6.

The product decision underneath

None of that tells you what your evidence actually needs, and that is the part only product can answer.

Masking is usually discussed as a compliance obligation. Treated well, it is a fidelity decision. The question is not "is this masked" but "which properties of the real data does this specific claim depend on?"

The fourth row is the one Episode 02 already warned about. Staging fixtures are clean, complete, and well-formed; production records contain partial migrations, legacy field formats, and account states the fixtures never modeled. A masking pipeline that produces tidy data has destroyed the exact property the evidence needed, and it will do so silently.

The cleanest implementation pattern is to never dump raw production data at all — export through a transformation query so unmasked data never lands in a non-production store6. That is the same structural-over-behavioral move the series has made five times: do not instruct the pipeline to delete the sensitive copy, arrange for it never to exist.

04Buying evidence without buying consequence

Four techniques, ordered by what they cost and what they can prove. The engineering conversation goes better when the AI PM knows which one answers which question.

Shadow

Production traffic is mirrored to a candidate system. The production system serves the user normally; the candidate's output is retained for comparison rather than returned to the user7. For agents, hiding the final response is insufficient if intermediate tools can still write, send, deploy, transact, or mutate state.

An agent is in shadow mode only when every consequential operation is intercepted before the system of record.

This is the highest-value technique in the set for agent work, and Episode 06 already depends on it. Shadow traffic reveals timeout patterns, retry amplification, and saturation points using real request flows rather than synthetic assumptions7.

Shadow preserves the input distribution, timing, and approved read paths while holding durable effects at zero. That combination makes it the default instrument for comparative decision evidence, not an advanced technique.

The failure mode is specific and must be checked rather than assumed. A shadow agent that writes to a real CRM, sends a real email, or calls a real payment API is not shadowing — it is running in production twice. Every tool must be intercepted at the environment boundary, structurally. Episode 03's enforcement argument applies unchanged: instructing the agent not to write is not the same control as the grant not existing.

Replay

Recorded historical traffic re-executed against a candidate. Record-and-replay captures real interactions and re-runs them8, and traffic replay platforms exist specifically to give production realism without production risk8.

Replay's advantage over shadow is that it is reproducible and can be run against known outcomes. Its limitation is temporal, and Episode 02 named it: a test suite running against the current system clock is not reproducible, because the relationship between the fixture and now silently changes. Replay requires the effective date to be pinned to the recorded date, or the eval measures the calendar rather than the agent.

Replay is also the only technique that can produce Episode 06's seeded cases honestly, because seeded cases must be sourced from historical genuine errors rather than synthesized.

Simulation

A constructed stand-in for the real dependency. The engineering discipline here is more mature than agent teams typically realize, and its vocabulary is worth borrowing.

Process simulation practice distinguishes tieback models giving simple output-for-input response, low fidelity adding directionally correct dynamic response, medium fidelity adding first-principles behavior that responds automatically to system actions, and high fidelity adding rigorous component-level modeling9. Digital twin practice adds the validation obligation: fidelity determines how well the twin mirrors real dynamics10, face validity is established subjectively while quantitative validation compares simulated output against real system data using statistical tests10.

Two transferable ideas. Fidelity can be deliberately decreased where the decision does not require it10 — a permission, not a compromise. And a simulation is itself an artifact requiring validation against the real system; otherwise the evidence measures the simulator.

For agents, the most common simulation is a stubbed tool. The under-asked question: does the stub reproduce the failure behavior of the real dependency — timeouts, partial results, rate limits, malformed responses — or only its success path? Episode 02 required declared degraded modes. A simulation that only succeeds cannot exercise them.

Game day

A scheduled exercise where failure is deliberately induced. Chaos practice supplies the method: define steady state, hypothesize it continues, introduce real-world failure variables, and look for a difference11. The governing constraint is blast radius — minimize it deliberately, start with the smallest scope that teaches something, and expand as confidence grows11.

This is the only technique that can satisfy Episode 06's seventh promotion condition, which requires demotion triggers to have fired in a drill before promotion takes effect. It is also the only way to discharge Episode 03's revocation requirement, which asks what a human can do in sixty seconds to stop a specific agent. Both are claims about behavior under failure, and neither can be evidenced by a system that has never failed.

Run game days in a test or staging environment first, then expand11; where they run against production, confirm with business stakeholders and hold a rollback plan11.

Which technique proves what

05Evidence-grade fidelity

The organizing principle, and the thing to put on a slide.

Every claim has a minimum fidelity below which the evidence does not support it, and a maximum above which you are buying risk that the claim does not require.

Specify fidelity from the claim, never from the availability of an environment.

Worked against the refund agent, dimension by dimension.

Three observations that carry the argument.

Only one row requires real consequence, and it is not in the table. No claim the refund agent needs to establish requires actions that take effect on real customers. Consequence fidelity is what promotion is for, not how promotion is evidenced — Episode 06's canary is the controlled introduction of consequence, staged and reversible, not a test.

The most valuable claims are shadow claims. Decision quality and saturation — the two hardest things to evidence and the two Episode 06 most depends on — need real data, real reach for reads, real scale, and real time, with consequence at zero. That is precisely what shadow provides.

Containment is the one claim requiring deliberate RC-3. Verifying that Episode 04's boundaries hold requires an environment where they can be tested against, which means real reach. This is where July 2026 lives, and it is the one place where the fidelity paradox cannot be dissolved — only bounded.

The evidence ceiling

An evidence ceiling is the strongest product claim an environment can support after accounting for every material difference between that environment and the world in which the agent will operate.

RC-4 does create additional evidence: downstream outcomes, customer response, and real operational behaviour. But it is no longer a harmless evaluation tier. It is production, and every artifact from Episodes 01 through 06 applies in full. If a team wants consequence fidelity, the correct instrument is Episode 06's canary — staged, reversible, and governed — not a realistic test environment.

06The test environment gets the same five artifacts

The structural claim of this episode, and the cheapest thing in it: nothing new needs to be invented. Episodes 01 through 05 already produced the governance an RC-3 environment requires. It simply was never pointed inward.

The third row deserves emphasis because it is counterintuitive and consistently true. An eval harness needs to provision, configure, inspect, and tear down environments, which tends to require broad infrastructure permissions. The evaluation control plane often holds broader authority than the agent under test. If the agent can cross into that control plane, the evaluation inherits the control plane's reach. In July 2026 the escape route ran through an internal service the harness depended on1.

The fifth row is the one that damages evidence quietly. Episode 05's core insight was that a system can degrade while every dashboard stays green. The same applies to fidelity: a masking pipeline that starts dropping edge cases, a stub that diverges from the real dependency, a data refresh that stops running. Nothing fails. The evidence simply stops meaning what it meant, and every claim resting on it becomes retroactively weaker without anyone being notified.

When fidelity drifts, old evidence does not disappear, but the claim it originally supported may no longer be defensible. Identify the affected versions and window, map them to dependent promotions, downgrade those claims to uncertain, replay representative cases where possible, and restore the gate when the missing evidence was load-bearing.

Canary tokens as the detection mechanism

One concrete control worth specifying, because it directly addresses the July 2026 failure mode at low cost.

A canary token is a decoy artifact designed to be attractive to an attacker12. Honeytokens sit at the data layer, embedded in real stores and indistinguishable from real data12. They detect intrusion during early stages such as reconnaissance or lateral movement, before critical systems are reached12, and function as lightweight tripwires12.

For a test environment the application is direct. Place decoy credentials that have no legitimate use anywhere in the system. Any use of one is unambiguous evidence that something inside the environment is doing something it was not asked to do — no anomaly detection, no threshold tuning, no false-positive budget. The signal has exactly one interpretation.

This is a structural detection control, and it belongs in the Containment Model wherever the threat model makes the decoy meaningful. A canary proves that one forbidden path was exercised. Its silence does not prove containment.

07What high fidelity costs

Fidelity is usually discussed as a technical constraint. It is a budget line, and the AI PM should be the one holding it.

The seventh row is where organizations balk, and the objection is understandable: full production governance on an environment that generates no revenue is a hard sell. July 2026 is the counter-argument, and it is available for free. The cost of governing that environment as RC-3 was a fraction of the cost of the disclosure, the forensic timeline, the credential rotation, and the third-party incident.

Fidelity debt

The mirror of Episode 06's autonomy debt, and it accumulates the same way.

Every evidence claim made on an environment less real than the claim required creates an obligation to either raise the fidelity or lower the claim. Organizations rarely do either. The debt surfaces when a production failure turns out to have been invisible to the test environment by construction — a data shape the masking removed, a dependency the stub always succeeded for, a scale the shadow never reached.

The tell is diagnostic and easy to check: when a production incident is reviewed, ask whether the test environment could have caught it in principle. A pattern of "no" answers is fidelity debt with a payment schedule.

08Translating this to your product

Five questions, whatever you build: What is the reachability class of every environment you test in, measured rather than named? Which claim requires the most fidelity, and on which dimension? Which dimension are you buying without needing it? Could your test environment have caught your last production incident? When did the masking pipeline last get reviewed against a schema change?

09The artifact

10Reviewing fidelity

Ninety minutes, with security, data governance, and the eval owner present.

Six questions per environment. What can it reach, verified against the allowlist rather than the label? What credentials exist inside its execution boundary? What is the strongest claim it supports, and are stronger claims being made? Does the harness hold broader authority than the agent under test? Would a canary token fire if something moved laterally? When was the masking pipeline last validated against a schema change?

Two questions for the whole plan. Could this register have caught our last production incident? Which claims currently in use rest on an environment that has since drifted?

11Anti-patterns

• "It's just a test environment." Classify by reachability. The label is not a control.
• "It's highly isolated." With limited access to one internal service, which is the sentence that precedes every escape1.
• "We use production data in staging so the tests are realistic." Test environments must never hold unmasked personal data from real users6.
• "The data is anonymised." If a mapping exists, it is pseudonymised and the regulation still applies5.
• "Masking is a compliance task." It is a fidelity decision. Tidy masked data destroys the edge cases the evidence needed.
• "We ran it in shadow." Were the write tools intercepted structurally, or was the agent instructed not to write?
• "Our evals are reproducible." Against the current clock? Then they measure the calendar.
• "The stub returns the right answer." Does it return timeouts, partial results, and rate limits? Degraded modes cannot be tested against a stub that always succeeds.
• "We tested the rollback." In a document, or in a game day?
• "The eval harness is internal tooling." It is more privileged than the agent it tests, and it is inside the blast radius.
• "Security reviews production." And the RC-3 environment nobody classified?
• "We'll do a game day after launch." Episode 06 condition seven requires the drill before promotion takes effect.
• "The masking pipeline has been running for a year." Through how many schema changes, reviewed by whom?
• "Higher fidelity is always better." Fidelity can be deliberately reduced where the decision does not require it10. Unneeded fidelity is purchased risk.

12What product owns

1. The environment register — every environment producing evidence, including the ad hoc ones nobody listed.
2. Reachability classification — measured against actual allowlists and credentials, not names.
3. The evidence ceiling per environment, and refusing claims that exceed it.
4. Which data properties the evidence depends on — the input to the masking specification, not an output of it.
5. Technique assignment — which claim is established by shadow, replay, simulation, or game day.
6. The interception requirement — that shadow write-blocking is structural.
7. Game day cadence and scope — which failures must be rehearsed, how often.
8. Fidelity drift signals and the retroactive rule that follows from them.
9. Evidence expiry conditions, and which Episode 06 promotions they affect.
10. Bringing test environments into production security review scope.

The envelope bounds what may happen. The manifest defines the world. The matrix establishes authority. The containment model states what survives compromise. The operating contract proves it all still holds. The autonomy ladder decides what that evidence has bought. The fidelity plan governs where the evidence came from — and whether it was ever worth what was claimed for it.

The line

A test does not become better by becoming more like production in every way.

It becomes better when it reproduces the production property the claim depends on — and refuses every real consequence the claim does not require.

Do not ask whether the environment is realistic.

Ask what it is realistic about, what it can still reach, and what its evidence is allowed to prove.

13The AI PM page

Print this. Bring it to the room.

Decisions you own

Ten questions

1. What can this environment actually reach, verified against the allowlist?
2. What credentials exist inside its execution boundary?
3. What is the strongest claim it supports, and are stronger claims being made?
4. Which fidelity dimensions are you buying, and against which claim?
5. Does the eval harness hold broader authority than the agent under test?
6. Are shadow write-blocks structural, or is the agent merely instructed?
7. Is the effective date pinned, or is the eval measuring the calendar?
8. Does the stub reproduce failures, or only successes?
9. When was the masking pipeline last validated against a schema change?
10. Could this register have caught our last production incident?

Red flags

• An environment described as isolated except for one thing.
• A test environment outside the production security review scope.
• Unmasked production data anywhere below RC-2.
• "Anonymised" used where a reversible mapping exists.
• Masked data that is tidier than production data.
• Shadow mode enforced by instruction rather than by absent grants.
• Evals running against the current system clock.
• Stubs with no failure paths.
• An eval harness sharing credentials with the agent under test.
• No canary tokens in any RC-2 or RC-3 environment.
• Game days that have never run, or ran more than a quarter ago.
• A promotion resting on evidence from an environment that has since drifted.

What you leave with

A complete environment register with measured reachability classes. An evidence ceiling per environment, and a list of claims currently exceeding it. A data fidelity specification stating which properties must survive masking. A technique assignment mapping every Episode 06 promotion condition to the environment that can evidence it. A fidelity drift signal list with a retroactive rule. A game day schedule with named scenarios.

One line to carry out: An environment that can reach a real system is a production environment — and what the organization calls it is not a control.

→Next · Make the Environment a Product

Seven artifacts now exist. Each answers one question a governance board, a security reviewer, or an auditor will eventually ask — and each was produced in a ninety-minute conversation rather than after an incident.

What remains is the question that decides whether any of it survives contact with an organization: who owns this, at what standard, and at what cost?

Episode 08 covers paved roads and golden environments, the release gate that assembles all seven artifacts into a single ship-or-hold decision, the per-run economics of environment provisioning, and the vendor questions that actually separate platforms once the marketing pages look identical. It also addresses the uncomfortable organizational fact underneath the whole series: every artifact here has an owner, and in most companies that owner does not currently exist.