The Harness Decision

The Story

A Series B fintech company raised $40M to build an AI-powered financial planning platform. Their pitch deck featured an "autonomous financial advisor" — an AI that could analyze a user's complete financial situation, generate personalized retirement scenarios, monitor market conditions, and adjust recommendations in real-time.

The engineering team designed the architecture to match the pitch deck. An orchestrator pattern: a central planning agent that delegates to specialized sub-agents — tax analysis, portfolio optimization, risk assessment, market monitoring. Each query triggered 4-7 model calls. The system was sophisticated, well-tested, and architecturally elegant.

It was also wrong.

When they launched to their first 200 beta users, the usage data told a different story. 73% of user sessions were simple, single-intent queries: "How much should I save for retirement if I want to retire at 60?" "What's the tax implication of converting my IRA to a Roth?"

These single-intent queries went through the full orchestrator pipeline. Total latency: 8-12 seconds. Total cost: $0.47 per query. The same quality answer could have been produced by a single model call with well-assembled context in 2 seconds for $0.03.

The team had built a race car for a commute that was 73% highway driving. The PM who approved the architecture had asked the right question — "What should our AI be able to do?" — but missed the more important question: "How much machinery does each user need actually need?"

The Core Idea

The harness decision is the most consequential architectural choice in an AI product. Not "which model?" Not "which framework?" The question is: how much harness?

Level 1: Single Prompt — assemble context, send to model, return response. No chaining, no routing. Cost: $0.01-0.05/query. Latency: 1-3 seconds. Appropriate more often than most teams admit.

Level 2: Chain — multiple model calls in a predetermined order. Each step's output feeds the next. Cost: $0.05-0.15/query. Latency: 3-8 seconds. Works when the task has a natural decomposition that's consistent across queries.

Level 3: Router — a decision point where different queries take different paths. Cost: $0.03-0.25/query (depends on route). Works when a product serves multiple distinct use cases.

Level 4: Orchestrator — the model decides what steps to take, executes them, evaluates results, decides next steps. Cost: $0.25-1.00+/query. Latency: 8-30+ seconds. Appropriate when the task genuinely requires dynamic multi-step reasoning.

Think of it like transportation for a 10-mile commute. You could walk (single prompt). Bike (chain). Drive (router). Hire a chauffeur with real-time traffic optimization (orchestrator). Most commutes need a bike. Some need a car. Almost none need the chauffeur. But the chauffeur FEELS like the right answer because sophisticated feels better than simple.

Anthropic's engineering team put the principle bluntly: avoid building agentic systems when simpler solutions suffice. They published five workflow patterns — prompt chaining, routing, parallelization, orchestrator-worker, and evaluator-optimizer — and explicitly ordered them from simple to complex, recommending that teams start at the simplest level and add complexity only when the simpler pattern demonstrably fails.

The over-engineering pattern is so common it deserves naming: complexity bias. Teams build orchestrators because orchestrators feel like real engineering. A single-prompt solution feels like a hack. The PM's job is to resist this. Complexity must earn its place.

The fintech company's resolution: they built a router in front of the orchestrator. Simple queries (73% of traffic) routed to a single-prompt path. Complex queries (27%) routed to the orchestrator. Average cost dropped from $0.47 to $0.14 per query. Average latency dropped from 9 seconds to 3.2 seconds. Quality was indistinguishable.

Where This Hits in Production

The cost multiplier is not theoretical. At 100,000 queries per day, the difference between single-prompt ($0.03/query = $90K/year) and orchestrator ($0.50/query = $1.5M/year) is $1.4M annually. That's headcount. That's runway.

Harness level should vary by query, not by product. A product isn't "a single-prompt product" or "an orchestrator product." It's a product with a distribution of query complexities. The harness should match the query, not the product's ambition.

Model capability shifts the spectrum over time. What required a 3-step chain in 2024 may require only a single prompt in 2026 because frontier models handle classification and retrieval-aware generation in one pass. The PM should revisit the harness decision every 6-12 months.

Remember This

1. The harness decision is a spectrum: single prompt → chain → router → orchestrator. Each level adds cost, latency, failure modes, and debugging complexity. The right answer is the simplest level that serves the actual usage distribution.

2. Cost differences are dramatic and real. Single prompt at $0.01/call vs. orchestrator at $0.50+/call is a 50x multiplier. At 100K queries/day, that's the difference between $90K/year and $1.5M/year.

3. Build from the bottom up. Start with a single prompt. Add complexity only when the simpler pattern demonstrably fails. And keep the router — so simple queries never pay the orchestrator tax.