Multi-Model Orchestration

The two products

Two AI products. Same workload class — long-form drafting with retrieval, light tool use, occasional multi-step reasoning. Same scale — roughly four million paid sessions a month. Same internal quality bar; both teams measure output quality with human pairwise comparisons against a frozen golden set, and both score within two points of each other on the rubric.

One product spends $4.80 per output session. The other spends $0.95.

The cheaper product is not using a worse model. It is using six different models, each chosen for a specific class of query. A 1.5B distilled router classifies the incoming request in under 40 milliseconds. Single-fact lookups go to a cached 7B open-weights model at fractional cost. Routine drafting goes to DeepSeek V4 Pro. Anything risk-tagged or flagged as nuanced reasoning goes to Claude Opus 4.7. Long-context summarisation goes to a long-context specialist model with prompt caching switched on. The handful of legal-review queries — less than 2% of volume — go to a multi-model ensemble that votes across three frontier models and returns only on consensus.

The expensive product sends every single query to one frontier model. Their team thinks of themselves as principled. We don’t compromise on quality. They are not compromising on quality. They are paying five times the necessary unit cost to deliver the same perceived quality, because 80% of their workload is queries the frontier model is wildly over-qualified for.

The cost difference compounds month over month into the only thing that ends up mattering at the board level: gross margin. The expensive product runs at 31% gross margin. The cheaper product runs at 78%. Both of them are growing. Only one of them survives the next funding cycle without an emergency price increase that breaks the customer promise.

This is the post about how the cheaper product got there, and why every serious AI product in 2026 will look like the cheaper one within twelve months — or be acquired, repriced, or shut down.

The frame

It is not a technical optimisation. It is not an infra concern. It is not the engineering team’s call. It is the product decision that lives upstream of every metric the business cares about — gross margin, sustainable price point, ability to expand into adjacent segments, ability to give it away free in a freemium tier. The PM who treats model selection as a one-time architecture choice rather than a continuous portfolio decision has surrendered the most powerful unit-economics lever they will ever have.

The 60-second answer

Every agent workflow has heterogeneous needs. Some queries are easy, some are hard. Some are latency-sensitive, some are not. Some need long context, some need raw speed. Sending all of them to a single frontier model pays four times the necessary cost on the 80% of queries the frontier model is over-qualified for.

The fix is multi-model orchestration: a router classifies each query and sends it to the right model for that class. Anthropic has named five architectural patterns for how this gets composed — chaining, routing, parallelisation, orchestrator-workers, and evaluator-optimiser. The discipline is to start with the simplest pattern that plausibly works and only escalate when evidence demands it.

A naive frontier-only deployment versus a well-designed multi-model stack with caching, batching and distillation is a 12–25x total cost difference at the unit-economics level — for the same or higher user-perceived quality. The PM owns this decision because the trade-offs are quality-versus-cost trade-offs, and quality-versus-cost is the PM’s job.

The 4× structural cost gap

The single most important fact about model pricing in 2026 is that the gap between the strongest open-weights and frontier closed-source has stabilised at roughly 4× — and the open-weights side is now genuinely good enough for the majority of production workloads.

DeepSeek V4 Pro completes the full Artificial Analysis Intelligence Index for $1,071 of inference cost, against Claude Opus 4.7 at $4,811 to run the same benchmark suite. That is a 4.5× cost gap, with the open-weights model losing only a few points on the hardest reasoning categories. DeepSeek’s API pricing sits at $0.14 per million input tokens and $0.028 per million for cached reads — roughly an order of magnitude below frontier-tier per-token pricing.

For most task surfaces — routine drafting, classification, extraction, summarisation, retrieval-augmented generation, customer-support reply suggestion — the cheaper model is acceptable. The user cannot perceive the difference on these queries because there is no perceivable difference; the task ceiling is below what either model is capable of.

For some task surfaces — high-stakes reasoning, tool-use chains with many steps, ambiguous-instruction following, novel domain transfer — the frontier model is genuinely better, and the user can feel the difference.

Routing the easy 80% of queries to the cheaper model and the hard 20% to the frontier is the difference between a 4× cost overhead on most of the workload and being competitive on price. A team that doesn’t route is paying frontier prices for cheap-model work; they are not buying any quality their users can detect, they are buying it because they never separated the two query classes in the first place.

This is the structural fact that makes multi-model strategy non-optional. The price gap is not closing. The frontier models will get better and stay expensive; the open-weights models will get better and stay cheap. The gap that exists today exists in 2027 and 2028. A product that is single-model in 2026 is making a choice that compounds against it for every quarter it stays single-model.

The five orchestration patterns

Anthropic’s Building Effective Agents paper names five architectural patterns that compose any non-trivial agent system. They are not five different approaches to the same problem; they are five different primitives that get layered, and the choice of which one to reach for is the architectural decision the PM lives with for the life of the product.

1. Chaining

A linear sequence of LLM calls where each call consumes the output of the previous one. Step one drafts, step two critiques, step three rewrites. Easiest to build. Easiest to observe. Easiest to debug because the failure mode is always traceable to a specific step. The right answer for most workflows. If the workload is decomposable into a fixed sequence of sub-tasks, chaining is what you want, and reaching for anything more complex is over-architecture.

2. Routing

An input gets classified, then sent down one of several specialised paths. This is the first lever a PM should reach for when introducing multi-model into an existing system. Trivial to implement — even a 50-line classifier improves cost dramatically. The unit-economics impact is large because the router lets you replace a single expensive model with a portfolio of right-sized models. Every serious AI product in 2026 has a router somewhere in the stack; teams that don’t are the teams paying 4× on the easy 80%.

3. Parallelisation

Concurrent LLM calls when sub-tasks are independent. If the user’s query needs three different things looked up that don’t depend on each other, you don’t run them sequentially — you fan them out, then aggregate. This is a latency win primarily, not a cost win; total tokens consumed are the same, but wall-clock time is roughly the longest sub-call instead of the sum. Useful for any workflow where user-perceived latency is a quality dimension.

4. Orchestrator-Workers

One LLM acts as the planner, dynamically decomposing the task and dispatching sub-tasks to worker LLMs. Powerful for genuinely open-ended agentic work. Operationally complex. Significantly more expensive than chaining because the orchestrator pays for every planning step, and the planner often re-plans mid-task. Reach for this pattern only when chaining cannot carry the workload — meaning the task structure genuinely cannot be pre-determined and has to be decomposed at runtime.

5. Evaluator-Optimiser

One LLM produces output; another LLM evaluates it and asks for revisions until it passes. Highest quality on the dimensions the evaluator measures. Highest cost because every output is being generated multiple times. Right pattern for high-stakes content where the cost of a bad output is much higher than the cost of two extra LLM calls — legal drafting, medical summarisation, executive-facing communication.

Anthropic’s stated discipline in the same paper is the line every PM should print and pin above their desk: start with the simplest pattern that plausibly works. Pattern over-architecture is the most common, most expensive failure mode in agent design in 2026. A team that uses orchestrator-workers when chaining would have worked has multiplied their cost, multiplied their failure modes, and made debugging materially harder — all for capability the user cannot perceive because the workload didn’t need it.

The right escalation path is sequential. Start with chaining. Add routing once you have enough volume to see the query-class distribution. Add parallelisation where latency hurts. Reach for orchestrator-workers only when you have a genuine planning problem that the simpler patterns cannot encode. Reach for evaluator-optimiser only on the high-stakes 1–3% of workload where it pays for itself.

Designing the router

Of the five patterns, routing is the one a PM should be most fluent in, because it is the one that delivers the largest unit-economics impact for the least architectural risk. The router is the strategic spine of the agent stack — the component that turns a portfolio of models into a coherent product.

There are four levels of router sophistication. Each level is the right answer at a different stage of the product’s life.

Level one — rule-based

Simple if-then logic on task type, risk level, query length, user tier, or any combination. If the query is under 50 tokens and contains no entity names, send it to the cheap model. If it’s tagged risk-high, send it to the frontier. If it’s a known intent class with high-confidence classification, send it to the specialist for that class. This is the right starting point for every team. It is fast to implement, fast to debug, and produces the bulk of the unit-economics gain that a more sophisticated router would produce. Most teams should stay here for the first six months of production.

Level two — distilled router

A small model — typically 1B to 7B parameters — fine-tuned on your own production traces to predict, for any incoming query, which downstream model will produce the best cost-quality outcome. Two to three times better than rule-based at the same latency budget, because it captures patterns the rules cannot articulate. The right move once you have enough production data to train it on — somewhere between 50,000 and 250,000 routed queries with labels. The training signal is the joint optimisation: did the chosen model produce acceptable quality, and did it do so at acceptable cost? Distillation generally produces 5–8× cost reduction with under 5% accuracy loss, and the same source documents one healthcare deployment dropping from $400K/month to $120K/month after applying distillation across the routing layer.

Level three — learned or reinforcement-learning router

A bandit or full RL setup that optimises for cost and quality jointly, with the reward signal being some weighted combination of user satisfaction, eval scores, and per-query cost. The highest long-term ceiling, because the router improves continuously from production traffic. Reach for it only when the distilled router has been operating long enough to provide a stable baseline and a clean training signal — typically a year of stable operation. The risk with RL routers is that the reward function is the entire product specification; if the reward is mis-specified, the router optimises for the wrong thing very quickly and very confidently.

Level four — multi-model ensemble router

Multiple models invoked in parallel for the same query, with voting or consensus determining the final output. This is the highest-quality, highest-cost pattern, and it should be reserved for the small fraction of workload where being wrong is catastrophic — legal review, medical decision support, financial compliance, anything customer-facing where a single bad output causes brand-level damage. A well-designed system uses ensembles for 1–3% of volume, not 100%.

The PM’s job is to know which level your product needs right now, and to resist the pressure from the engineering team to skip ahead. Skipping from rules straight to RL is the architectural over-build trap restated in router form. The team that runs solid rule-based routing for six months learns more about their query distribution than the team that built a learned router on day one ever will.

Orchestration patterns layered on routing

Once the router exists, four further patterns determine what happens once a query has been classified.

Cascading

The cheap model attempts the query first; the system measures confidence; if confidence is below threshold, the query escalates to a more expensive model. The cheap model handles the queries it can handle; the expensive model handles only the queries the cheap model couldn’t. Excellent unit economics when the confidence signal is reliable. Brittle when confidence is poorly calibrated, because over-confident wrong answers from the cheap model never escalate.

Parallel voting

Multiple models receive the same query simultaneously; their answers are aggregated by voting, consensus, or a downstream judge model. Higher cost than cascading because every model runs every time. Higher quality because disagreements between models surface uncertainty the system can act on. Right pattern for high-stakes workflows where the cost of consensus is acceptable.

Speculative decoding

A fast draft model generates a candidate output; a stronger verifier model checks it token-by-token and only re-generates the segments where it disagrees with the draft. Latency win because most tokens are generated by the fast model; quality win because the strong model has the final say on every token. Operationally subtle to get right; needs the two models to be similar enough in vocabulary that the strong model can verify the draft without rewriting it from scratch.

Hierarchical routing

A high-level router classifies into broad categories; each category has its own specialist router that handles the fine-grained classification. Useful when the query distribution has clear top-level structure — a customer-support agent might first route by language, then within each language by intent. Adds an extra hop of latency but makes each individual routing decision cleaner because the classifier at each level only has to discriminate within a narrower set of options.

These four patterns compose with the five Anthropic primitives, not replace them. A production system will typically use chaining as the outer skeleton, routing to choose models for each step, parallelisation where steps are independent, cascading inside the routing decisions, and evaluator-optimiser on the final output for high-stakes flows. The architecture is layered, and each layer is a specific cost-quality decision.

Mixture of Experts at the orchestration layer

Mixture of Experts (MoE) began as a model-architecture pattern — multiple specialised sub-networks behind a learned gating function, only a fraction of the parameters firing on any given token. The idea generalises one level up the stack. The orchestration-layer analog of MoE is routing across multiple specialised external models rather than running one general-purpose model on every query.

The orchestration-layer MoE pattern earns its complexity in three situations. Domain specialisation — a fine-tuned legal model handles legal queries, a fine-tuned medical model handles clinical queries, a general model handles everything else. Modality specialisation — a vision model owns image-heavy tasks, a code model owns code generation, a general model owns text. Cost optimisation — matching model size to query complexity dynamically, which is the SLM-plus-frontier pattern restated as a two-expert MoE.

The trade-off is the same trade-off that exists at every layer of this article: complexity is not free. Orchestration-layer MoE adds routing logic, model-management surface area, and observability burden across multiple paths. It is justified only when the cost or quality lift is large enough to absorb the operational complexity — and for most portfolios, that means the simple two-tier SLM-plus-frontier escalation captures the bulk of the savings before the team even needs to think about specialised experts.

The full cost-optimisation stack

Multi-model is the optimisation that multiplies the impact of every other lever. Routing is the global optimiser; the levers it composes with are local optimisations that each compound on top.

Prompt caching at 90% discount on cache reads means that any system prompt, retrieval context, or persistent instruction that gets reused across queries is paid for at one-tenth the rate of a fresh read. A router that sends similar queries to the same model lets caching work; a router that scatters queries randomly across models destroys caching’s effectiveness, because each model has its own cache.

Anthropic’s Batch API offers a 50% discount on input and output tokens for asynchronous workloads, and OpenAI’s Batch API has the same 50% discount. For any workload that does not need a real-time response — overnight summarisation, reprocessing pipelines, eval runs, training data generation — batching is free money the team is leaving on the table if they aren’t using it.

Self-hosted inference with vLLM’s PagedAttention and continuous batching drives per-token cost on open-weights models down by another two to four times below API pricing, when volume is high enough to justify the operational overhead. Continuous batching means the GPU is never idle waiting for the slowest sequence in a batch to finish; new requests slot into the batch as soon as any existing request completes a token.

Distillation — training a smaller model on the outputs of a larger one — produces a model that handles a specific task surface at fractional cost with minimal quality loss. The 5–8× reduction figure is conservative for narrow tasks; for tasks that are well-defined and high-volume, the gap can be larger.

These levers compose multiplicatively, not additively. A workload that routes 80% of volume to a 4× cheaper model, runs 60% of that volume through cache reads at 90% discount, runs 40% as overnight batch at 50% discount, and replaces the most common task surface with a distilled specialist at 5× reduction — that workload is somewhere between 12× and 25× cheaper than the same workload sent naively to a single frontier model with no caching, no batching, and no distillation. This is not theoretical; it is the actual cost gap between the two products in the opening scene, and it is the gap that determines which AI products survive 2026.

The five traps

The patterns above are well-documented and well-understood. The reason teams still get this wrong is that the failure modes are PM-owned, not engineer-owned, and PMs new to the AI stack default into them quickly. Across the case studies that have surfaced in 2026, portfolios that move from frontier-everywhere to multi-model orchestration capture 60–80% inference cost reduction with no measurable quality degradation — and the reason most teams leave that money on the table is one of the five traps below.

These five traps are not engineering oversights. They are PM oversights, because the PM is the one who decides whether the team measures query distribution before routing, whether escalation to complex patterns requires evidence, whether observability is built before the router ships, whether evals gate every routing decision, and whether MoE earns its complexity. None of these decisions are made by engineers without PM authority; all of them are abandoned when the PM is not paying attention.

Remember this

1. Cost / Capability / Speed Decision Triangle. Three dimensions determine model fit. Most workloads do not need all three at maximum; the orchestration job is to match each workload to the right corner.
2. SLM-plus-frontier escalation is the default pattern. Roughly 70% routine to a small model, 30% complex to the frontier, with low-confidence fallback — ~63% cost reduction at the same quality bar.
3. MoE at the orchestration layer is the second optimisation, not the first. Specialised experts earn their complexity only after the two-tier router has been measured.
4. 60–80% inference savings is the typical portfolio capture. The number is consistent across case studies that move from frontier-everywhere to multi-model orchestration.
5. Multi-model orchestration is a cost-strategy decision, not engineering plumbing. At portfolio scale, model selection has unit-economics, vendor-strategy and risk-management implications — PM ownership is non-negotiable.

In practice

A six-step playbook for moving a portfolio from frontier-everywhere to a deliberate multi-model stack.

1. Run the portfolio model-spend audit. Where is the inference budget concentrated? In most portfolios, the top three initiatives consume 70–90% of spend. The L1-T08 cost-attribution dashboard surfaces this.
2. Score top initiatives on the Decision Triangle. For each initiative, score complexity, latency tolerance, and reliability requirement. Match model tier to the score — not to the team’s default.
3. Implement the SLM-plus-frontier escalation pattern. Complexity classifier in front, cheap-model path for the routine majority, frontier path for the hard cases, low-confidence fallback. Quality monitoring on both paths from day one.
4. Decide on MoE patterns where applicable. Domain-specialised models, modality-specialised models — only where the volume and capability gap justify the operational complexity. Track the operational cost as carefully as the inference cost.
5. Run quarterly orchestration reviews. Model tiers shift as model capabilities evolve. The right tiering for Q1 is rarely the right tiering for Q4. Re-score, re-route, retire stale specialists.
6. Translate orchestration savings into business language. “Multi-model orchestration on the top three initiatives reduced portfolio inference spend by 67% over four quarters, recovering X gross-margin points without quality degradation.” This is the sentence that earns the discipline its budget for the next year.