- 01. How the context assembly pipeline works at production scale -- parallel execution, failure isolation, and caching across stages.
- 02. Why funnel-mode retrieval (100 candidates to 5 selections) is the dominant production pattern, and the latency breakdown of each stage.
- 03. How to design assembly as a pipeline of independent, testable stages that can fail without taking down the system.
- 04. Why optimizing assembly latency without measuring assembly quality is the single most expensive mistake AI teams make.
The Story
A commercial insurance platform processes 40,000 underwriting queries per day across 200 enterprise accounts. Each query requires the system to assemble a context window from six sources: tenant underwriting guidelines, property data, historical claims, regulatory requirements, conversation history, and available tools for risk calculation.
In v1, the assembly was sequential. Load the system prompt. Call the property database. Embed the query and search the knowledge base. Look up conversation history. Load tool definitions. Format the output template. Total assembly time: 1,400ms before the model even started generating.
But the performance problem masked a worse problem: fragility. When the property database had a slow response, the entire assembly pipeline stalled. Five stages waited in line behind one slow database.
The engineering team rebuilt assembly as a parallel pipeline with independent stages. Assembly dropped from 1,400ms to 380ms. Tuesday afternoon spikes stopped taking down the product.
The Core Idea
Principle 1: Parallel execution with dependency graphs. Not all assembly stages depend on each other. In practice, a production assembly pipeline has two phases. Phase 1 (parallel): constitution loading, intent classification, memory fetch, and static tool loading all fire simultaneously. Phase 2 (dependent): knowledge retrieval and example selection depend on the classified intent from Phase 1.
The wall-clock time drops from the sum of all stages to the maximum of the parallel group plus the maximum of the dependent group.
Hidden Latency Tax.
51% faster. At 40K queries/day, that's 3.6 million ms saved -- per day.
Principle 2: Funnel-mode retrieval. The knowledge retrieval stage is almost always the bottleneck. Production systems use a three-stage funnel that progressively narrows candidates: broad vector search (100 candidates), cross-encoder reranking (down to 20), and strategic selection with business logic (final 5).
The quality difference between "top 5 by embedding similarity" and "top 5 after reranking" is substantial -- reranking improves retrieval precision by 15-30% in production benchmarks.
Principle 3: Stage-level failure isolation. Every assembly stage will fail eventually. Production assembly treats each stage as an independent unit with its own fallback. Constitution fails -- serve from local cache. Knowledge retrieval fails -- operate without RAG, include a disclaimer. Memory fails -- treat as new user. The user gets a slightly less personalized response, not an error page.
Tenant isolation in assembly. In B2B products, context assembly has to load different constitutions, different knowledge bases, different tool permissions, and different memory stores per tenant -- from the same infrastructure. A misconfigured tenant ID that loads Account A's constitution for Account B's query is not just a quality failure -- in regulated industries, it is a compliance violation. Tenant isolation typically uses a tenant configuration layer that resolves during Phase 1, before any dependent stages fire.
Assembly caching. Constitution, tool definitions, and tenant config rarely change between queries. A well-designed caching layer can reduce assembly's effective latency by 30-50% for repeat queries. The key discipline: cache invalidation. When a tenant updates their configuration, every cached constitution for that tenant must expire immediately. Stale cache in assembly means the model sees outdated rules -- a silent, dangerous failure.
Are Often the Wrong 5.
The top-5 by embedding distance are often the wrong 5.
Where This Hits in Production
Latency budgets are product decisions. The PM sets the total latency budget (say, 3 seconds for customer-facing chat). Assembly must fit within 300-600ms. If the PM specifies "always rerank with a cross-encoder," that's a 50-200ms commitment. If the product can't afford it, the PM trades assembly quality for latency -- a product tradeoff, not an engineering one.
Assembly is your largest monitoring surface. When a production AI product gives bad answers, the debugging question is almost never "was the model wrong?" It's "what did the model see?" Assembly-level tracing turns "the AI was wrong" into "retrieval returned stale documents because the index hadn't been updated since Tuesday."
Token budget enforcement happens at assembly time. The context window has a practical budget -- not the model's maximum, but the quality threshold from Context Rot. Assembly enforces this by trimming lower-priority content when the window exceeds the limit. The trimming priority order -- never trim the constitution or the user message; trim oldest memory first, then lowest-ranked knowledge chunks, then secondary examples -- is a product specification the PM defines, not an engineering default.
Connecting the Dots
Each assembly stage maps to one or more layers of the CONTEXT framework: Constitution loading maps to C, knowledge retrieval to kNowledge, memory fetch to Tracks, tool selection to Equipment, template application to Template. The assembly pipeline IS the CONTEXT framework in operation.
The relationship to the harness. Assembly is the core of the harness. The harness includes monitoring, economics, failure handling, and lifecycle management -- but assembly is what runs on every query. If the harness is the cockpit, assembly is the primary flight instruments.
The eval connection sharpens here. A retrieval eval tests whether the right documents were retrieved -- but only assembly-level tracing can tell you why the wrong documents were retrieved. Was the query embedding poor? Was the vector index stale? Did the reranker mismatch? Did a tenant configuration error route to the wrong knowledge base? See Evaluating the Harness for the full pipeline discipline.
In Practice -- Assembly Trace with Latency and Token Accounting
Here is a production assembly trace for a single query in a B2B customer-support platform -- every stage, its latency, its token contribution, and the parallel-execution structure.
The query. An enterprise customer's IT admin types: "Why is our SSO integration failing for users in the EMEA region?"
ASSEMBLY TRACE -- Query #2,847,103
Tenant: acme-corp (Enterprise) User: it-admin-markov@acme.com
Query: "Why is our SSO integration failing for users in the EMEA region?"
================ PHASE 1 -- PARALLEL (no dependencies) ================
CONSTITUTION LOAD 8ms 2,400 tokens
Source: cache (tenant config v12.4)
Acme Corp custom rules; EMEA data residency acknowledged
INTENT CLASSIFICATION 54ms
Model: Haiku Intent: technical_troubleshoot
Sub-intent: sso_integration_failure Confidence: 0.96
MEMORY FETCH 72ms 1,240 tokens
Profile: IT Admin (senior, self-service) 280
Recent 3 interactions (cert rotation, EMEA…) 620
Episodic (Feb '25 SSO outage -- SAML refresh) 340
TOOL LOADING 11ms 630 tokens
Loaded 4 of 46 tools for technical_troubleshoot
Phase 1 wall-clock: 72ms (bounded by memory fetch)
================ PHASE 2 -- DEPENDENT (needs intent) ==================
KNOWLEDGE RETRIEVAL (funnel mode)
Stage 1 -- Broad retrieve 67ms 93 candidates from 2 indexes
Stage 2 -- Rerank 148ms 93 -> 18 above 0.65 threshold
Stage 3 -- Select 12ms 5 chunks (3,800 tokens)
[1] SSO SAML troubleshooting guide (0.94, 3 days old)
[2] EMEA region IdP config requirements (0.91, 12 days old)
[3] Known issue: Azure AD EMEA cert (0.88, 1 day old)
[4] SSO error code reference (0.83)
[5] Acme Corp integration spec (tenant) (0.79)
Retrieval total: 227ms
EXAMPLE INJECTION 9ms 420 tokens
TEMPLATE APPLICATION 4ms 120 tokens
Phase 2 wall-clock: 227ms (bounded by retrieval)
================ ASSEMBLY COMPLETE ====================================
TOKEN BUDGET Used % Max
-----------------------------------------------
Constitution 2,400 28% 3,000
Knowledge 3,800 45% 5,000
Memory 1,240 15% 2,000
Tools 630 7% 1,000
Examples 420 5% 600
Template 120 1% 200
User message 32 <1% 500
-----------------------------------------------
TOTAL 8,642 12,300 WITHIN BUDGET
ASSEMBLY LATENCY
Phase 1 (parallel): 72ms
Phase 2 (dependent): 227ms
Validation + append: 3ms
-----------------------------------------------
TOTAL ASSEMBLY: 302ms (budget 500ms)
Sequential equivalent: 614ms -- 51% parallelism savings
What the IT admin sees. A structured diagnosis that references the known Azure AD certificate propagation issue in EMEA (chunk #3), connects it to the admin's recent cert rotation question (memory), provides specific troubleshooting steps from the SSO guide (chunk #1), and notes that a similar issue was resolved in February via SAML metadata refresh (episodic memory). Response time: 2.1 seconds.
What the PM should see in this trace. Knowledge retrieval consumes 75% of the assembly latency. The reranking stage alone is 148ms -- justified by the quality improvement: chunk #3 (the known issue posted yesterday) would have ranked 8th by embedding similarity alone, but rose to 3rd after reranking because of recency and specificity. The tenant-specific document (chunk #5) was retrieved because the assembly pipeline queries a tenant-isolated index. The PM who can read this trace knows exactly which stage to invest in next sprint and exactly what trade-off they are making.
The Trap
Optimizing assembly latency without measuring assembly quality.
A team removes reranking to save 180ms. Assembly drops from 420ms to 240ms. Over the next two weeks, answer quality drops 8%. The team investigates the model, checks for prompt regressions. They don't look at assembly because "we only improved it."
The fix: every assembly optimization must be evaluated against both latency AND retrieval quality metrics. Latency without quality is a faster path to wrong answers.Remember This
1. Context assembly is a parallel pipeline, not a sequential list. The wall-clock latency is the maximum of the parallel group plus the maximum of the dependent group -- not the sum of all stages.
2. Funnel-mode retrieval (broad retrieve, precise rerank, strategic select) is how production systems get the right 5 chunks. Skipping reranking trades 15-30% retrieval precision for 50-200ms of latency. That's a product tradeoff the PM must own.
3. Every assembly stage will fail. Design each stage with an independent fallback that degrades the experience rather than breaking it.
References
1. Building Effective Agents -- Anthropic Engineering Blog
2. Context Engineering for AI Agents -- Anthropic Engineering Blog
3. Towards Efficient RAG: A Survey -- arXiv, January 2025