Contain the Compromise

Vocabulary for this episode

Coined here: Compromise ledger (the artifact). Failure domain (everything that fails together when one credential, policy bundle, deployment, or administrator fails). Parameter provenance (where each consequential value is permitted to originate). Declared trust anchor (a compromise the model explicitly does not bound, with a named owner).

What you own

This is the episode most likely to be handed entirely to security, and the one where that handoff costs most. Security can build every control here. Security cannot decide what the business is willing to lose.

Security builds containment. You decide what is worth containing, and what it is acceptable to lose.

Where this sits

EPISODE 01  Bounds what one run may cause
EPISODE 02  Defines the world it reasons in
EPISODE 03  Establishes whose authority it carries
        |
EPISODE 04 — THIS ONE
What still holds when all three are
under attack, and where it stops
        |
EPISODE 05  Whether all four hold in production

This episode introduces few new controls. It is a stress test of the previous three artifacts. The envelope's ceilings, the manifest's egress lists, and the matrix's grants are all claims. This episode asks which are still true when the agent is working for someone else.

Every "no" is a finding. Most teams discover that between a third and a half of their controls are preferences.

01When the agent cannot be trusted

The instinct is to ask which controls "survive compromise." That question is too vague to answer, and answering it loosely produces false confidence.

No control is structural in the abstract. A gateway ceiling survives a compromised model; not a compromised gateway. A sandbox survives malicious generated code; not a kernel escape. An authorization service survives a leaked grant; not a stolen signing key.

A control is structural only in relation to a stated compromise.

Name the failure first, then judge the control against it. Three properties decide it.

Externality. Is the control enforced outside the component you assume is compromised?

Independence. Does it share code, identity, configuration, policy bundle, deployment pipeline, or administrator with the failed component?

Fail behavior. When unavailable or unsure, does it deny, degrade, or quietly allow?

The third is most often left undefined, and it decides whether a control is a boundary or a formality. A gateway that lets requests through when its policy service times out is enforcing a ceiling on days when the network is healthy.

Against a compromised model, any control the model can reason its way past is a preference, not a boundary.

Behavioral controls are not worthless. OWASP lists constraining model behavior, separating and labelling external content, and privilege control among legitimate mitigations, alongside human approval for high-risk actions. The point is ordering: behavioral controls reduce how often compromise happens; structural controls bound what it costs. Only the second belongs in a release gate.

02Define the trusted boundary

"Compromised reasoning" bundles at least six failures, each invalidating a different set of controls.

Decision integrity is lost when the system can no longer trust the action the agent chose, the target it chose, the parameters it supplied, or its account of what it did.

A defense can preserve one and lose another. An architecture stopping untrusted content from redirecting the agent's steps can still let it corrupt an amount the agent extracts. Section 05 develops this.

Trust zones

• Untrusted — model output, retrieved content, customer text, merchant notes, attachments, generated code, subprocesses, tool descriptions.
• Enforcement — authorization service, payment gateway, egress proxy, sandbox supervisor, mail gateway.
• Administrative — grant issuer config, policy deployment, identity revocation, kill switch.
• Evidence — append-only pipeline, external retention, incident timeline.

A control survives only if both the place it is enforced and the place it is administered sit outside the compromise you assumed.

The administrative half carries most of the weight and is the half teams forget. A proxy outside the agent container is useless if the agent can reach its config socket, call an unauthenticated admin endpoint, or edit its policy file. Sitting outside the process is necessary, not sufficient.

03Map the exposure

Simon Willison's lethal trifecta: an agent becomes acutely dangerous when it combines access to private data, exposure to untrusted content, and the ability to communicate externally. Exfiltration needs no dramatic channel — outbound requests, generated links, and image loads all qualify.

That framework is about confidentiality, and it is three conditions, not four. Extending it silently would misrepresent it. But the same structure applies to three kinds of loss.

The third row is missing from most agent threat models and is often the easiest attack to run. An agent need not be tricked into stealing anything to cause an outage. It only needs to be tricked into looping.

Blast radius has three dimensions: what the agent can expose, what it can change, and what it can consume.

This is where Episode 03's read-authority pairing becomes visible: widening one column silently widens the whole row.

04Assume specific compromises

"Assume breach" is a Zero Trust principle — design on the assumption attackers are already operating inside, focusing on limiting impact and enabling fast detection. For agents that must be written out compromise by compromise, and the list must extend past the model.

Naming the last two as declared trust anchors rather than pretending to bound them is what makes the rest credible. A containment model that claims to survive everything has not been thought through, and a reviewer stops trusting the document at the point they notice.

Correlated failure

The obvious rule — two controls in the same system are one control — is correct and far too weak. Controls in entirely different services still fail together if they share a cloud account, IAM role, signing key, policy bundle, deployment pipeline, identity provider, administrator, or one malformed field.

Two services do not make two controls. If one credential, policy bundle, deployment, or administrator can disable both, they share a failure domain.

The first two rows look independent and are not. One bad policy deployment removes both. That is a finding, and surfacing findings like it is why the ledger exists.

Count only the controls the compromised component cannot reach, then subtract those sharing its failure domain. What remains is your actual depth.

05Separate content from authority

The strongest research direction does not harden the model. It restructures the system so untrusted content never reaches the component holding authority.

The dual-LLM pattern separates two models. A privileged model holds tool access and never sees untrusted input. A quarantined model processes untrusted data and has no tool access. They communicate through symbolic variables — the quarantined model returns a reference, the privileged model acts on it without reading the tainted content.

CaMeL makes this enforceable. It builds a protective layer that extracts control flow and data flow from the trusted query, so untrusted data can never change the program's flow, and uses capabilities to block exfiltration over unauthorized data flows. A privileged model emits code describing control flow; an interpreter runs it while tracking a data-flow graph with capabilities on each variable; policies are checked at tool-call time.

Untrusted content may influence what the agent knows. It must never influence what the agent is permitted to do.

What this does not solve

CaMeL reports 77% of AgentDojo tasks solved with provable security against 84% for the undefended baseline. That gap is the product trade-off — securing control flow costs capability, and someone decides whether the trade is acceptable. That someone is you.

The paper is also explicit that CaMeL offers no way to make a sequence of instructions atomic, and no rollback for operations that failed after causing side effects. A half-finished sequence that already moved money is exactly Episode 01's idempotency case.

And protecting control flow does not make untrusted data true.

Authorization can constrain an action without proving its parameters are correct.

Parameter provenance

Suppose a poisoned document states the disputed amount is ₹1,847 when the actual transaction was ₹184.70. The agent is authorized to refund T17. The ceiling holds. The case binding holds. Every control passes. The customer receives the wrong refund, and no permission was violated.

No consequential parameter may originate in untrusted content. Either it is looked up from an authoritative system, or the action does not proceed.

This connects prompt injection to ordinary data quality, where most AI PMs already have strong instincts.

06The tool arrived from somewhere

Every artifact so far treats tools as given. The envelope lists them. The manifest declares them. The matrix grants authority through them. The ledger has a row for a tool misbehaving at runtime.

None ask where the tool came from, who wrote its description, or whether it is the same tool that was approved.

MCP has made this urgent. When an agent connects to a tool server it reads tool metadata — names, descriptions, parameter schemas — to discover what it can do. That metadata enters the model's context as instruction .

Three distinct attacks

Tool poisoning. Malicious instructions embedded in a tool description, invisible to a user reading a tool list and fully visible to the model reading metadata. The agent is compromised during capability discovery — before a single tool is called .

Rug pull. A tool registers legitimately, is reviewed and adopted, then later changes its description or quietly gains a capability. The agent reads the new description and adopts the new instructions. The review approved a version that no longer exists .

Tool shadowing. A tool positions itself to intercept calls meant for a legitimate one, or its description influences how the agent uses a different tool it was never granted authority over .

Why this breaks the previous three episodes

Read the matrix row carefully. Supply-chain compromise does not defeat your grants. It steers the agent toward the most damaging thing your grants allow. That is an argument for tighter grants, not evidence the grants failed.

What contains it

The last row is the one that survives. Everything above reduces the chance of poisoning. Only the last bounds the cost, because a grant does not widen because a tool description asked it to.

The pinning row maps onto something Episode 02 already argued: pin the container image by digest, not tag, because a tag can be repointed at different content without changing the thing referencing it. A tool description is the same problem one layer up. The name stays constant; the content changes.

A tool description is executable content arriving from outside your organization and entering the model's context as instruction. It belongs in the manifest — pinned, versioned, re-approved on change.

What you own: which tool servers this workflow may use at all; whether a description change requires re-approval before the next run; whether tool metadata is treated as trusted or untrusted.

The question: "If the description of an approved tool changed tomorrow morning, what would happen, and who would know?" If the agent would silently adopt it, you have an unpinned dependency in the most privileged position in the system.

07Bound action and egress

Egress is a capability, not a network setting

Deny-by-default egress with an allowlist covering only what the task requires. The refinement that matters: treat every allowed domain as a capability grant rather than a destination, because every function reachable through it becomes attack surface.

Allowing api.payments.internal grants every endpoint on that host the agent's credential can reach. An allowlist entry is a permissions decision wearing networking clothes — which is why it belongs in this review, not a networking ticket.

Where the boundary sits

Anthropic's Claude Code sandboxing makes placement explicit: filesystem isolation permits reads and writes inside the working directory while blocking modification outside, and network traffic leaves through a Unix domain socket to a proxy running outside the sandbox. The boundaries cover not only the agent's own actions but scripts, programs, and subprocesses spawned by a command, with the OS enforcing the boundary for every command and its children.

Subprocess coverage is the detail most homegrown sandboxes miss. A boundary you escape by spawning a child process is not a boundary.

State the guarantee honestly. Anthropic's own documentation notes isolation reduces impact without eliminating risk: permitted egress can still leak readable data, and a writable project directory still allows code modification. The network sandbox has had reported bypasses, including a hostname-handling flaw permitting blocked traffic and exfiltration, since patched.

That history supports this episode's thesis.

A sandbox is a containment layer, not a proof of containment.

Which is why the ledger requires a second, independent control for every row where sandbox escape is the assumed compromise.

Why each grant property is separately necessary

A grant with a ceiling but no resource binding permits unlimited ₹2,000 refunds across every customer. A compromised agent is best understood as a live channel for both data and action: it holds credentials, reads internal systems, and has outbound access.

The capability that does not exist

The refund agent holds no credential authorizing subscription changes. Not restricted, not approval-gated. None was ever issued.

Every capability you provision must survive the compromise test. Every capability you decline to provision passes it automatically.

Third appearance of the same principle — Episode 01 as reach, Episode 02 as the absent executable, Episode 03 as the grant that does not exist. Cheapest control in the series, most often traded away for convenience during a sprint.

08Bound the fleet

Every control so far bounds a single run. The ledger can be complete, every row can have an independent surviving control, and the system can still fail catastrophically.

A thousand concurrent runs, each holding a valid ₹2,000 grant, all compromised by the same poisoned template: ₹20 lakh exposure before anyone revokes anything — while every individual run stays within policy.

A bounded run can still produce an unbounded fleet.

Version matters because agent compromise is often correlated by construction. A poisoned template, bad policy bundle, changed tool description, or compromised dependency affects every run of that version at once. Runs that look independent but share a version share a failure domain.

Resource containment

A compromised agent that can neither exfiltrate nor transact can still consume a queue, exhaust a rate limit, or burn a budget until the surrounding service degrades. Episode 01's run budget was a cost control. Under compromise it becomes a containment control — worth pointing out to a team that resented it.

09Detect and interrupt

Containment bounds damage. It does not end an incident. Four phases, and merging them is a common design error:

• Limit — what bounds damage before anyone notices?
• Detect — what makes the compromise visible?
• Interrupt — what stops further action?
• Recover — how is trusted operation re-established?

A system can contain without detecting, and detect without being able to stop. Both are real failure modes with different owners.

Denials are telemetry

Agent compromise looks like ordinary operation: each action within policy, the pattern clearly wrong. The most reliable signal is the one your structural controls already generate for free.

A denied grant, a blocked egress attempt, a failed parameter validation — each is not only a prevented harm but evidence the agent is reaching for things the envelope forbids. A system that denies silently has thrown away its best detection signal.

That is the opening scene restated as a design principle. Nobody detected the compromise. The refusals did.

Response classes, not invented numbers

Thresholds like "two denied grants in one run" are easy to write and have no empirical basis. Publishing them is worse than publishing nothing, because teams copy tables. What can be specified is the response class.

The first four are categorically different: under a correctly specified task they cannot happen at all, so one occurrence suffices. The rest need calibration because legitimate variation exists.

10Count the cost of halting

Same discipline as Episodes 01–03, and it matters more here, because the cost of a false halt is paid by a customer mid-transaction.

What does a false positive cost? A halted run mid-refund is not neutral. It leaves a customer waiting and a case ambiguous.

Who absorbs the halted work, how fast? A halt is a transfer to a human queue. If nobody watches that queue at 2am, the halt is a silent abandonment.

What if this fires a hundred times in an hour? Either you have an incident or a miscalibrated threshold, and the correct responses are opposite. The system should tell you which.

A halt is not free and it is not neutral. It converts a security event into an operations event, and someone has to be standing there when it does.

The failure mode to design against is the one that kills every alerting system: a threshold so tight that operators disable it. A control switched off during the first busy week is worse than a looser control that stays on, because everyone believes it is still running.

Interruption

Alert with enough context to triage, confirm scope from traces — how many sessions, one tool, one prompt version, one customer segment — then choose the least disruptive intervention that works, keeping a full halt for when nothing narrower contains the harm. The kill switch must be a deterministic, infrastructure-level control operated from a control plane the agent cannot administer.

A stop the agent has to cooperate with is not a stop.

Target response times belong here as service-level objectives proven in drills, not numbers asserted in a document.

Human approval, qualified

OWASP recommends human-in-the-loop for privileged operations. An approval step is not automatically a boundary. It survives model compromise only when:

• Authoritative view — the reviewer sees source data, not the model's summary.
• Exact intent — the approved intent names exact target, parameters, amount.
• Single use — short-lived, non-reusable.
• Immutable — no parameter changes between approval and execution.
• Verified downstream — the gateway checks the approved intent, not that some approval exists.
• Sustainable load — reviewers are not flooded into rubber-stamping.

Human approval is a boundary only when the human approves the authoritative action intent, not the agent's persuasive description of it.

Episode 03 asked whether the approver holds the authority. This episode asks whether they still have the attention. Both fail silently, in the same direction.

11Recover cleanly

Evidence under compromise

Saying a compromised evidence sink is survived by "append-only, external retention" is circular. Decompose it.

The last row belongs in the artifact. An audit architecture claiming assurance when every trust anchor has failed is not describing a real system.

Resume, or fork

Episode 02 established a resumed run must not assume the world is unchanged. Episode 03 added it must not assume its grant went unused. Under compromise the hardest case appears: a snapshot taken after injection faithfully restores the injected state. Snapshot-and-rehydrate, a resilience feature in Episode 02, becomes a liability.

Never resume from state whose integrity is uncertain. Terminate the compromised execution. Recovery may fork from a verified pre-compromise checkpoint, and only after external side effects are reconciled, all credentials replaced, tainted context excluded, and the work explicitly re-authorized.

A fork of a recovered execution, not a resumption of a compromised one. The distinction forces the four conditions to be met explicitly rather than assumed.

12Memory outlives the run

Everything above bounds a run. Episodes 01–03 bound a run. The run is the unit of the entire series.

Persistent memory breaks that unit, invisibly.

If the agent writes to a memory store, vector index, case summary, or learned-preferences table that a later run reads, compromise is no longer bounded by the run. Poisoned content written at 14:22 Tuesday is authoritative context for a clean run on Friday.

OWASP tracks memory and context poisoning as a distinct threat for exactly this reason: attackers compromise the stored and retrievable data agentic systems depend on, ingested from uploads, API feeds, user input, or peer-agent exchanges .

Why every existing control misses it

The compromised run was contained perfectly and still left a persistent artifact. Containment succeeded and the compromise survived it.

What the model must specify

Two rules do most of the work.

Memory is untrusted on read. Content written by a previous run is agent-generated content, and §05 established agent output cannot supply a consequential parameter. It does not become trustworthy by being stored.

Memory carries the run ID that wrote it. When a run is declared compromised, every entry it wrote is identifiable and revocable. Without provenance tagging your only options are to trust everything or purge the store — and under incident pressure teams choose the first.

Containment that bounds the run but not what the run wrote down is containment with a delay fuse on it.

What you own: deciding whether this agent needs persistent memory at all. In a surprising number of workflows the answer is no, and it was added because it demoed well. The cheapest containment control available is a memory store that does not exist.

13Who declares a compromise?

The entire second half of this episode — halts, forks, credential rotation, re-authorization — assumes someone decided a compromise occurred.

That role appears nowhere in the artifact, and in most organizations it does not exist.

Without it: a structural denial fires at 02:40. An on-call engineer sees a blocked egress attempt. It could be compromise, a misconfiguration after yesterday's deploy, or a customer pasting something odd into a form. The engineer lacks authority to halt a revenue-affecting fleet and lacks context to rule out a false positive. So they file a ticket.

The decision gets made at 09:15 by whoever reads it first.

Four decisions needing a named owner

Two are security decisions and two are product decisions — and not the two people expect.

Declaring a compromise is a security call. They have the signal expertise.

Choosing the intervention rung is a product call, because the trade-off is containment against availability, and only product knows what a four-hour halt costs relative to a bounded exposure.

Authorizing the fork is joint. Security confirms the state is clean; product confirms the work should resume.

Declaring closure is a product call, because it means accepting residual risk.

The question: "At 2am on a Sunday, who has both the authority and the information to halt this fleet?" If they would first have to wake someone else up, the containment model has a gap no control in it addresses.

14The clock you did not know was running

Recovery in §11 is entirely technical. A parallel process starts at the same moment and has a legal deadline.

If the compromise involved personal data, GDPR Article 33 requires the controller to notify the supervisory authority without undue delay and, where feasible, not later than 72 hours after becoming aware of it, unless the breach is unlikely to result in a risk to the rights and freedoms of natural persons. Later notification must be accompanied by reasons for the delay .

The notification must describe the nature of the breach including, where possible, the categories and approximate number of data subjects and records concerned, the likely consequences, and the measures taken .

Read that as a design specification, because that is what it is.

To notify within 72 hours you must state, from evidence, which data the agent could read during the compromise window, which was actually accessed, how many data subjects that touches, and whether any left. Reconstructed by hand during the incident, 72 hours is not enough, and the reasons-for-delay clause becomes your problem.

The row most teams fail is "what it actually read." Writes are logged everywhere because they change things. Reads frequently are not, because they are cheap and voluminous — so during an exfiltration assessment you cannot distinguish what the agent could have read from what it did, and must assume the worst.

That assumption is the difference between notifying about forty data subjects and notifying about forty thousand.

The 72-hour clock starts when you become aware, not when you finish investigating. Everything you cannot reconstruct in that window, you must assume against yourself.

What you own: knowing which regimes apply, and making the evidence design good enough to answer a scoped question rather than a worst-case one. A one-hour conversation with legal that can only usefully happen before an incident.

15Translating this to your product

Five questions, whatever you build: Which control survives if the model is compromised? Do any two "independent" controls share a policy bundle? Which consequential parameter could come from untrusted content? What does the fleet version of this failure cost? Who declares the compromise?

16The artifact

17How to run the review

Ninety minutes, with security present and one person assigned to argue the attacker's side.

Four questions per compromise row: Which component are we assuming is compromised? Which independent trust anchor remains, and does it share a failure domain? What is the maximum credible loss before interruption — per action, per run, across the fleet? How is trusted operation re-established?

Supplementary probes. Which consequential parameters could originate in untrusted content? Is egress enforced outside the sandbox, and does it cover subprocesses? If one grant leaked right now, which property bounds the loss? Are structural denials logged as telemetry or discarded? What halts a single run without the agent's cooperation? Who authorizes a new run afterward? What happens if an approved tool's description changes tomorrow?

18Anti-patterns

• "Our system prompt tells it not to." Behavioral. Fails exactly when it matters.
• "We filter for injection patterns." Reduces frequency, not consequence. OWASP is explicit that current approaches cannot fully prevent injection.
• "It runs in a container." Not a sandbox if egress is open or subprocesses are unconstrained.
• "The allowlist is in the tool code." That code sits inside the compromised zone.
• "The proxy is external, so we're covered." Check the administrative path, and check whether the sandbox has a bypass history.
• "We have five layers." Count only those independent of the failed component, then subtract those sharing its failure domain.
• "Two different services, so two controls." Not if one policy bundle, key, or administrator disables both.
• "Each run is capped at ₹2,000." A thousand concurrent runs are capped at ₹20 lakh.
• "The tool is from an approved registry." Approved when? Pinned to what?
• "The action was authorized." Authorization bounds the operation, not the truth of its parameters.
• "A human approved it." They approved the agent's description, unless the interface showed authoritative source data.
• "The run was contained." And what did it write to memory?
• "We can turn it off." From where, how fast, and does it need the agent's cooperation?
• "Security will handle the incident." Who chooses the intervention rung?
• "We'll resume after investigating." Restoring a post-compromise snapshot restores the compromise.

19What product owns

Security owns sandboxes, proxies, and control planes. Ten decisions are not theirs.

1. Exposure acceptance — which intersections the agent carries, and why each cannot be removed.
2. Scope boundary — which compromises this model bounds, and which are declared trust anchors with named owners.
3. Tool admission — which servers this workflow may use, and whether a description change blocks the next run.
4. Parameter provenance — which values must come from authoritative systems.
5. Memory — whether it exists, what may write to it, how it is trusted on read.
6. Fleet bounds — maximum aggregate exposure across concurrent runs and versions.
7. Response classes — which signals warrant an immediate halt, and who absorbs the halted work.
8. Intervention targets — what must be stoppable, at what granularity, within what time.
9. Decision rights — who declares a compromise, chooses the rung, authorizes the fork, and declares closure.
10. Disclosure readiness — which regimes apply, and whether the evidence supports a scoped assessment.

The envelope bounds what may happen. The manifest defines the world. The matrix establishes authority. The containment model states what the system still guarantees when all three are under attack — and where those guarantees stop.

The line

A control is structural only in relation to a stated compromise.

Two services do not make two controls.

A bounded run can still produce an unbounded fleet.

Authorization bounds the operation, not the truth of its parameters.

A sandbox is a containment layer, not a proof of containment.

Containment that bounds the run but not what the run wrote down has a delay fuse on it.

→Next · Prove the Environment Is Working

Four artifacts now govern the run. Together they describe a system that is bounded, attested, accountable, and defended. They do not tell you whether it is working.

This episode's signals were tuned to catch an adversary — structural denials, egress attempts, provenance violations. Those fire when something is attacking.

The next episode asks the harder question: what tells you the system is quietly getting worse when nobody is attacking it?

A refusal rate is not a failure rate. A completion rate is not a success rate. A rising escalation count might mean the agent is degrading, or that the boundaries are working exactly as designed. Telling those apart is the difference between an operable fleet and an expensive one.

Episodes 01–04 determine what the system may do. Episode 05 determines whether you can tell what it did.

You now hold Artifact 04 — the Containment Model. The next question If nothing is attacking, how would you know it is degrading?

20The AI PM page

Print this. Bring it to the room.

Decisions you own

Ten questions

1. Which control survives if the model is compromised?
2. Do any two "independent" controls share a policy bundle, key, or admin?
3. Which consequential parameter could originate in untrusted content?
4. Is egress enforced outside the sandbox, and does it cover subprocesses?
5. If a grant leaked right now, which property bounds the loss?
6. What does this failure cost multiplied by concurrency?
7. If an approved tool's description changed tomorrow, who would know?
8. What does a compromised run write that outlives it?
9. At 2am Sunday, who has authority and information to halt the fleet?
10. Could you scope a disclosure assessment in 72 hours from existing logs?

Red flags

• The ledger has no rows marked out of scope.
• "Independent" is asserted, never traced to a shared dependency.
• Nobody can name who receives a halted run at 2am.
• Tool servers are approved once and never re-checked.
• Memory exists and nobody can say what writes to it.
• The kill switch has never been tested.
• The DPO has never seen this document.

What you leave with

A completed compromise ledger with at least two out-of-scope rows and named owners. A parameter provenance table. A fleet exposure number. A refusal-cost owner per halt condition. A decision rights table with real names. A tested revocation path. A disclosure readiness assessment.

One line to carry out: A control is structural only in relation to a stated compromise.