The Autonomy Design

"Can" and "Should" Are Different Decisions

In 2024, a mid-size insurance company deployed an AI agent for claims processing. The system could pull customer records, cross-reference policy details, calculate settlement estimates, draft correspondence, and initiate payment transfers. The team set the agent to Level 3 across the board -- act with monitoring. The reasoning made sense on a whiteboard: the company trusted the model, the claims adjusters were overloaded, and the whole point was reducing human bottlenecks.

For three weeks, it worked beautifully. Response times dropped from four days to six hours. Customer satisfaction scores climbed.

Then the agent sent a $340,000 settlement letter to a commercial policyholder for a claim that had been flagged for potential fraud investigation. The letter included specific dollar amounts, an acceptance deadline, and language that -- according to outside counsel -- constituted a binding offer. The fraud flag was a data point in the context, not a hard constraint in the agent's boundary design.

The failure wasn't technical. The model was working correctly. The failure was a design error: treating the entire agent as one autonomy level. Pulling a customer record and sending a six-figure settlement letter are both "actions." They demand wildly different levels of human oversight.

Designing the Autonomy Envelope

Within a single agent, different capabilities need different amounts of freedom. The capability-level autonomy framework has five levels:

Level 0: Suggest Only. Agent drafts output, surfaces it. No action taken. Human decides AND acts. For irreversible actions with high consequence.

Level 1: Suggest with Explanation. Agent drafts output, explains reasoning, recommends action. Human decides with context.

Level 2: Act and Report. Agent executes, then immediately notifies the user. Human reviews after the fact. For reversible actions with moderate consequence.

Level 3: Act with Monitoring. Agent executes, action is logged, human reviews periodically. For low-consequence actions where speed matters.

Level 4: Act Autonomously. Agent executes, no notification unless anomaly detected. For trivial, fully reversible, high-volume actions.

Think of it like a hospital's authority matrix. An attending physician can prescribe controlled substances. A resident needs sign-off for controlled substances. A nurse can administer but can't prescribe. Same hospital, same patient care system -- but the authority level for each action maps to the consequences of getting it wrong.

Where Autonomy Limits Get Negotiated, Quietly

The "same action, different context" problem. Sending an email is not one action. Sending an internal status update to a colleague is low-consequence, reversible, and well within the agent's capability. Sending an external settlement offer to a claimant's attorney is irreversible, high-consequence, and requires legal judgment. The same verb -- "send email" -- maps to two different autonomy levels depending on the recipient, the content, and the downstream implications.

Regulated industries and the hard ceiling. Healthcare, financial services, defense, and government face mandatory Level 0 requirements for specific actions regardless of model reliability. A clinical decision support agent cannot autonomously modify a treatment plan. A banking agent cannot autonomously execute trades above certain thresholds. The boundary matrix must encode these regulatory constraints as immovable floors.

The monitoring economics. Level 3 (act with monitoring) sounds efficient until you calculate the monitoring cost. If the agent performs 2,000 actions per day and the monitoring dashboard surfaces all of them, no human is actually reviewing 2,000 line items. Real Level 3 requires anomaly-based monitoring: the system surfaces only actions that deviate from expected patterns.

Writing the Autonomy Spec

Consider a Fortune 500 manufacturer deploying an HR agent fielding 400-600 requests per day across a 35,000-person workforce. The team's first instinct was to make the agent a "Level 2 self-service system." The HR director pushed back: "You're telling me the agent handles a PTO balance lookup and a disability accommodation request at the same autonomy level?"

That question produced the boundary matrix. Information retrieval runs at Level 3-4 because it's perfectly reversible. Routine transactions sit at Level 2-3. HR communications drop sharply -- a benefits explanation email that misrepresents coverage could cause real harm. Sensitive operations are all Level 0 by regulatory mandate.

The result: the agent handles 70% of daily requests at Level 3-4, 20% at Level 1-2, and 10% at Level 0. The HR team's workload dropped by 60%. Six months in, PTO request submission moved from Level 3 to Level 4 after 3,200 successful executions with zero errors. Accommodation requests stayed at Level 0. Some capabilities don't graduate.

Remember This

1. Autonomy is a per-capability decision, not a per-agent decision. The same agent should operate at Level 4 for database reads and Level 0 for financial commitments. One level for the whole agent is the single most common design error in production agent systems.

2. The boundary matrix maps every capability to an autonomy level using four factors: reversibility, consequence magnitude, user trust, and organizational risk tolerance. It's a deliverable the PM owns and updates as the agent's track record evolves.

3. The matrix is not static. Capabilities earn higher autonomy through demonstrated reliability against predefined graduation criteria. Start restrictive. Promote with evidence. Never promote on intuition or executive pressure.