Agent Reliability

The 90% Trap: Why Agents Fail in the Long Tail

In 2025, a major coding agent achieved 49% on SWE-bench Verified -- solving roughly half of real-world GitHub issues. The industry celebrated. But a senior engineering director at a Fortune 100 financial services firm looked at that number differently. Her team was evaluating agent-assisted code migration -- moving 2,400 microservices from Java 11 to Java 21. Each migration involved 8 discrete steps.

She did the math. If the agent succeeded at each step with 90% reliability, the probability of completing all 8 steps without error was 0.9^8 = 43%. Less than half of migrations would complete autonomously. At 95% per step: 66%. At 99% per step: 92%. The gap between 90% and 99% is 9 percentage points per step. The gap in end-to-end outcomes is 43% to 92%.

She approved the project with a condition: measure per-step reliability independently, identify the weakest step, and invest there first. That decision saved the project. Step 5 (resolving compilation errors) had only 72% reliability. That single step dragged end-to-end to 28%. Improving step 5 to 95% lifted overall performance to 59%. One step improved. Overall success nearly doubled.

Per-Step Reliability Compounds Multiplicatively

The formula is simple. The implications are not. If each step in an agent's workflow succeeds with probability p, and the workflow has k steps, the probability that ALL steps succeed is p^k.

The reliability stack. Pass^k applies not just within the agent but across the entire Agentic Stack. Context reliability at 95% across 7 layers gives 69.8%. Harness reliability at 99% across 7 steps gives 93.2%. Agent reliability at 90% across 10 steps gives 34.9%. System reliability = 0.698 x 0.932 x 0.349 = 22.7%.

Where to invest. Step 1: Measure per-step. Step 2: Find the weakest link. Step 3: Invest at the weakest step. The options, ranked by typical effectiveness: better context assembly, retry with modified approach, better tool definitions, step decomposition, and model upgrade as a last resort.

The Math Your Demo Hides From You

The 90% illusion. Sales decks quote "90% accuracy" as if it's impressive for an agent. For single-step operations, it is. For a 10-step workflow, 90% per step delivers successful results 35% of the time. Would you ship a product that fails 65% of the time?

Reliability as competitive moat. The team that achieves 95% per step first has a product that works twice as often as the competition's at 90%. That's not an incremental advantage. That's a category-defining gap.

Designing for End-to-End Pass Rate

A production customer service agent at a financial services firm runs a 7-step workflow. End-to-end success: 58.4%. Per-step measurement revealed Step 7 (Execute Action) at 88.6% was the bottleneck. Three targeted fixes -- auth token refresh, API timeout increase, and tool description rewrite -- projected to lift end-to-end from 58.4% to 73.2%.

Remember This

1. Pass^k: 90% per step x 10 steps = 35% end-to-end. 95% per step x 10 steps = 60%. That 5-point per-step improvement nearly doubles end-to-end success. Returns INCREASE as you approach perfection.

2. Always measure per-step, not just end-to-end. Find the weakest step. Invest there first. The weakest step dominates the chain's reliability.

3. Reliability compounds across the entire stack. A product "good" at every layer can still deliver poor outcomes because compound probability is merciless.