Agent Governance

Three Lists Every Agent Needs Before Launch

In February 2024, Air Canada's customer service chatbot told a grieving customer that he could book a full-fare flight and later apply for a bereavement fare discount retroactively. The customer booked the flight. Applied for the discount. Air Canada denied the claim -- their actual policy didn't allow retroactive bereavement applications.

The customer sued. Air Canada argued that the chatbot was "a separate legal entity" responsible for its own statements. The Canadian Civil Resolution Tribunal ruled otherwise: the chatbot is Air Canada's agent, and Air Canada is liable for its representations regardless of whether a human or an AI made them.

The fine was modest -- roughly $800 in fare difference plus tribunal costs. The precedent was not. A Canadian tribunal had established that companies are legally responsible for what their AI agents tell customers, even when the AI is wrong.

This case was a chatbot. Not even an agent. It followed scripted flows and occasionally generated natural language responses. No tools, no loop, no autonomy. Now extrapolate to agents that send emails, modify database records, execute financial transactions, and file insurance claims. Each action is a commitment the organization makes. Governance isn't a nice-to-have. It's the infrastructure that separates "AI product" from "liability engine."

Permitted, Required, Prohibited

Agent governance answers three questions at runtime. Not in a policy document. Not in a design review. AT RUNTIME -- in code, enforced by the harness, with no exceptions.

Question 1: What CAN the agent do? Capability boundaries. The exhaustive list of tools, data sources, and actions available to the agent. The boundary is defined positively -- anything not explicitly on the list is prohibited. This is the allowlist model, not a blocklist. Blocklists fail because you can't enumerate everything the agent shouldn't do.

Question 2: What MUST the agent do? Mandatory behaviors. Log every action taken. Explain every decision made. Escalate when confidence falls below a threshold. Check compliance constraints before executing any external action. Each "MUST" is a harness-level middleware -- code that runs before or after every agent action, regardless of what the model decided to do.

Question 3: What must the agent NEVER do? Hard constraints. Never disclose another customer's data. Never recommend discontinuing prescribed medication. Never commit the organization to a binding agreement without human approval. A violated "NEVER" is a catastrophe -- the agent did something that causes real harm.

Think of it like a hospital's operating room protocols. The surgeon has discretion about how to perform the procedure. But patient identity must be verified before cutting (MUST), only approved instruments on the tray (CAN), never operate on the wrong limb (NEVER). The protocols aren't suggestions -- they're physical infrastructure that enforce themselves regardless of the surgeon's judgment.

Kill switches. Three types, each for different failure modes. Hard kill: immediate termination, all agent activity stops. Used for catastrophic failures. Soft circuit breaker: gradual degradation, autonomy reduced to suggestion-only mode. Used for quality degradation. Budget kill: resource-based termination when the agent has consumed its allocated tokens, time, or API calls.

Progressive deployment. The staged rollout that makes agent deployment survivable. Stage 1 -- Shadow mode: agent runs in parallel, outputs never shown to users, 2-4 weeks. Stage 2 -- Supervised: agent handles 5-10% of real traffic, every action reviewed by a human. Stage 3 -- Bounded: 25-50% of traffic, human review only for high-risk actions. Stage 4 -- Full autonomy: all traffic, human review by exception only.

When Governance Becomes the Bottleneck

Governance as compliance evidence. In regulated industries, the governance infrastructure IS the compliance evidence. When an auditor asks "how do you ensure your AI doesn't make unauthorized financial commitments?" the answer is the CAN/MUST/NEVER matrix, the kill switch architecture, and the audit trail. If these are policy documents, the auditor's next question is "how do you enforce them?" If they're code, the audit trail speaks for itself.

The governance tax. Governance adds latency. Every MUST-do adds processing time. A typical governance stack adds 50-200ms per action -- negligible for a single action, significant across a 10-step workflow (0.5-2.0 seconds). The PM budgets this overhead explicitly: governance latency is a feature cost, not a bug.

Multi-jurisdiction complexity. A global enterprise agent must enforce different governance rules for different jurisdictions. The EU agent cannot store personal data beyond the session. The US agent must retain certain records. The governance infrastructure must be parameterized by jurisdiction.

Writing the Three Lists With Legal in the Room

Consider a procurement agent at a manufacturing company handling purchase requisitions, vendor comparisons, and order placement. The CAN list: search approved vendors, compare prices, generate purchase requisition drafts, query inventory levels, place orders under $5,000. The MUST list: log every action, include 3 vendor options (anti-single-source policy), check budget, verify requestor authority, apply sustainability scoring.

The NEVER list: place order from non-approved vendor (hard kill), exceed budget allocation (budget kill), split orders to circumvent approval thresholds (hard kill -- order splitting is a fraud indicator). The progressive deployment moved from shadow mode (weeks 1-2, less than 5% decision divergence from human baseline) through supervised (weeks 3-4) and bounded (weeks 5-8) to full autonomy within the $5,000 limit.

Remember This

1. Governance answers three runtime questions: what CAN the agent do (allowlist of capabilities), what MUST it do (mandatory behaviors like logging and compliance checks), and what must it NEVER do (hard constraints with catastrophic consequences). All three must be enforced as code in the harness, not as instructions in the system prompt.

2. Kill switches come in three types: hard kill (immediate termination), soft circuit breaker (autonomy reduction), and budget kill (resource cap). Deploy all three. Each catches a different failure mode.

3. Progressive deployment -- shadow, supervised, bounded, full autonomy -- with specific metric thresholds for each stage transition, is how you make agent deployment survivable. Skip stages and you'll discover governance gaps in production, at full blast.