Knowledge at Inference

The Story

In 2023, ChatGPT fabricated six court decisions for a New York attorney named Steven Schwartz — complete with court names, docket numbers, and crisp legal reasoning. He filed the brief. Opposing counsel checked the citations. None of the cases existed. Every one was a plausible-sounding fiction.

The judge sanctioned Schwartz and his firm. The incident became a cautionary tale about AI hallucination — and it is still cited every time someone asks why AI systems lie with confidence.

The deeper lesson is not about hallucination. It is about knowledge architecture.

Schwartz was using a model with no access to legal databases. No connection to case-law repositories. No retrieval system whatsoever. He was relying entirely on the model's parametric knowledge — the patterns the model picked up during training. (Parametric knowledge — what the model "remembers" from being trained on a snapshot of public text. Useful for general reasoning. Unreliable for specific facts. Almost useless for anything published after the snapshot date.) For legal citations, that knowledge is unreliable in a particular way: the model does not store cases. It stores patterns of what cases sound like. So when asked for a citation, it produces something that looks like a citation. Court name. Docket number. Year. Plausible argument. Often complete fiction.

The fix is not a smarter model. The fix is a knowledge pipeline — a system that retrieves actual case law from a verified database and injects it into the context window before the model generates. The model's job shifts from recall the right case (which it cannot do reliably) to reason about the case I have been given (which it can). That shift — from knowledge-from-memory to knowledge-at-inference — is what RAG makes possible.

(Source: NY Times — Lawyer Sanctioned for ChatGPT Hallucinations)

The Core Idea

RAG — Retrieval-Augmented Generation — is the most widely adopted pattern in AI product development. (Retrieval-Augmented Generation — fetch the right document at the moment of the question, and slip it into the model's input alongside the question. The model now reasons over a real document, not its memory of similar documents.) The concept is simple: instead of relying on the model's training data, retrieve relevant documents from an external knowledge base and inject them into the context window so the model reasons over actual information rather than its parametric approximation.

Here is where teams go wrong. They treat RAG as a product feature. "We added RAG" goes on the quarterly roadmap. Marketing writes about "AI powered by your data." Engineering ships a vector database, connects it to the model, and moves on.

RAG is not a feature. It is the kNowledge layer of the seven-letter framework from The CONTEXT Stack — an information supply chain that runs before every model answer. Like any supply chain, its quality depends on every link, not just the final delivery.

The chain has seven stages. Naming them separately matters because each stage fails differently and requires a different fix.

Stage 1 — Ingestion. Your documents enter the system. PDFs, web pages, databases, APIs, knowledge bases — raw source material. The quality ceiling is set here. If the wrong documents go in, or the right documents in the wrong format, every later stage inherits the problem.

Stage 2 — Chunking. Documents get split into smaller segments — chunks — because the model cannot process an entire 200-page manual in one shot. Fixed-size chunks (split every 500 tokens) are simple but can cut a sentence, a paragraph, or a legal clause in half. Semantic chunks (split at natural boundaries — sections, headings, topics) preserve meaning but require more engineering.

Stage 3 — Embedding. Each chunk gets converted into a numerical fingerprint called an embedding. (Embedding — a list of numbers that captures the meaning of a piece of text, so that similar concepts end up with similar numbers.) These embeddings live in a vector database, ready for comparison.

Stage 4 — Retrieval. When a user asks a question, the question gets embedded too, and the system finds the chunks whose embeddings are closest to the question's. This is where semantic search happens — matching by meaning. A query about "sugar problems" should match a document about "diabetes mellitus" even though the words do not overlap.

Stage 5 — Reranking. The first retrieval step typically returns 20-50 candidate chunks. A reranker — a smaller, specialised model trained for relevance ranking — re-scores these candidates and picks the top three to five most relevant. Research shows reranking improves retrieval accuracy by 20-35% over retrieval alone.

Stage 6 — Injection. The selected chunks get formatted and placed into the context window at the right position. From The Context Window you already know that position decides whether the model attends to information. The relevant knowledge belongs near the end of the context, close to the user's question — not buried in the middle, where the model's attention is weakest.

Stage 7 — Generation. The model produces its answer using the injected knowledge.

Think of it like an expert analyst preparing for a briefing. They do not rely on what they remember from grad school. They go to the filing cabinet (retrieval), pull the relevant folders (ranking), select the key pages (reranking), lay them on the desk in the right order (injection), and then synthesise the briefing (generation). Their expertise is in synthesis. Not in memorisation.

Seven stages mean seven places to fail. In production, the failure is almost never at stage 7 (generation). It is upstream.

Retrieval miss. The system does not find the right document. The user asks about a specific policy, the embedding does not match, and the model generates from its parametric knowledge — which may be wrong. The fix: hybrid search — combining vector similarity with old-fashioned keyword matching. Hybrid search reduces misses by 15-67% over either method alone, because it catches both semantic matches ("sugar problem" → diabetes) and exact matches (policy number, product code, proper name).

Ranking error. The right document is retrieved but ranked below irrelevant ones. The model sees five chunks; the relevant one is chunk four; the model pays more attention to chunks one through three because of the attention pattern from The Context Window. The fix: aggressive reranking with a quality threshold. If the best match scores below a confidence threshold, return "I don't have enough information" rather than generating from poor-quality context. Refusing to answer beats answering wrong.

Staleness. The knowledge base contains outdated information. The user asks about the 2024 return policy; the 2022 version outranks the 2024 version because it has more text about returns. The model confidently cites the wrong policy. This is a metadata problem: chunks need timestamps, version numbers, and freshness scores that affect ranking.

Context-memory conflict. The retrieved document says one thing; the model's training data says another. Research shows models exhibit parametric bias — they tend to favour what they learned in training over what is provided in context, especially for popular facts. This is one of the most dangerous failure modes because the model sounds confident while ignoring its own context.

The hallucination paradox. RAG reduces hallucination. It does not eliminate it. Legal RAG systems marketed as "hallucination-free" still hallucinate 17-33% on case citations because the model sometimes generates plausible-sounding extensions or modifications of the retrieved text. The model is not lying. It is doing what language models do: generating probable next tokens based on context. Sometimes the most probable next token is a fabrication that sounds consistent with the retrieved facts.

Not all knowledge belongs in RAG. A useful taxonomy for the PM:

• Parametric knowledge (from training): general world knowledge, language patterns, reasoning ability. Always available, sometimes wrong, never current.
• Static documents (monthly update): policy manuals, product catalogues, procedure guides. Classic RAG territory.
• Indexed content (daily update): knowledge-base articles, FAQ updates, recent news. RAG with freshness management.
• Real-time data (by the minute): stock prices, inventory levels, system status. Not RAG — API calls into the Equipment layer (covered in Tools Shape Behavior).
• User-specific data (per user): account information, transaction history, preferences. Not RAG — the Tracks layer (memory, covered in Memory Architecture).

The PM's architectural decision: which knowledge type serves each user question? A product that routes everything through RAG when some questions need real-time APIs and others need user-specific data is the supply-chain equivalent of shipping every package via the same courier. Same delivery method for everything. Wrong tool for most jobs.

Where This Hits in Production

Multi-tenant knowledge isolation. In B2B platforms where one product serves many enterprise customers, Customer A's policy documents must never appear in Customer B's context window. This is not just a retrieval filter — it is a security architecture that has to be enforced at every stage: separate document collections per tenant, tenant-tagged embeddings, tenant-filtered retrieval, audit logging that proves isolation. A single cross-tenant leak is a data breach, not a relevance problem.

Knowledge freshness as compliance. In financial services, serving a customer a response based on an outdated interest rate or a superseded regulation is not just a bad experience. It is a compliance violation. The Constitution from The Constitution may commit the product to "never provide outdated financial information." The kNowledge layer has to enforce that commitment: freshness metadata on every chunk, date-aware ranking, automatic eviction of expired content.

Knowledge poisoning. Research demonstrated that just five carefully crafted documents injected into a RAG knowledge base can manipulate responses 90% of the time — the PoisonedRAG attack. In enterprise contexts where users contribute to the knowledge base, this creates a real threat: a malicious contributor can shape AI responses for every other user by inserting targeted documents that rank highly for specific queries. The PM has to design knowledge ingestion with validation, source tracking, and anomaly detection. Not just relevance scoring.

Connecting the Dots

Here is the reframe that changes how practitioners think about RAG: it is not a product feature you ship once. It is an ongoing quality discipline — closer to data quality in the data-engineering world than to feature release in product management.

The eval connection. The RAG Triad — three metrics that diagnose the supply chain at different stages — is introduced in detail in evals: Context relevance (did you retrieve the right documents?), Faithfulness (does the response accurately reflect those documents?), Answer relevance (does the response actually answer the question?). The dangerous pattern these expose: high faithfulness, low answer relevance. The model accurately summarised the document it was given — but it was given the wrong document. The faithfulness score of 0.96 looks great. The answer relevance of 0.63 reveals the real problem. Average them to 0.82 and the failure hides.

On the autonomy spectrum. RAG's role transforms as autonomy rises. At Levels 1-3, the harness controls retrieval — the PM designs what to retrieve and when. At Levels 5-7, the agent decides what knowledge it needs and retrieves it autonomously. The supply chain becomes a loop, not a pipeline. The full autonomy framing lives in Spectrum of Agency.

The moat. A great RAG pipeline is harder to copy than the model behind it. Anyone can rent the same model. Nobody can rent your knowledge base — your years of curated documents, your domain-specific embeddings, your tenant-specific freshness rules, your reranker trained on your real production failures. RAG is institutional knowledge encoded into infrastructure.

In Practice

Here is what RAG management looks like when it is treated as a supply chain rather than a feature.

Consider a commercial insurance company whose AI assistant helps underwriters evaluate risk. (Underwriting — the function inside an insurance company that decides whether to insure a particular customer or property, and at what price.) The knowledge base includes policy manuals, regulatory filings, claims history data, and industry risk reports. The audit maps every knowledge type to the right delivery mechanism — RAG for policy manuals, API tools for real-time claims data, the Tracks layer for account-specific history, parametric knowledge for general industry terms. Then the team instruments every stage and watches the dashboard weekly.

The dashboard reveals something counter-intuitive: retrieval precision looks good (91.4%), but end-to-end accuracy fails (88.3%). The gap is explained by the failure analysis. Staleness accounts for 31% of wrong answers. The right documents are being found — but the wrong version of the right documents.

That is the working discipline of the kNowledge layer — supply-chain thinking, not feature thinking. Map every knowledge type to the right delivery mechanism. Instrument every stage. Trace every failure to the specific link that broke. Fix the link. Repeat. Quality compounds. Retrieval becomes a real moat, not a slide.

Remember This

1. RAG is a seven-stage supply chain, not a feature toggle. Each stage fails differently and requires a different fix. Retrieval miss, ranking error, staleness, context-memory conflict, and generation hallucination are distinct problems with distinct solutions.

2. Not all knowledge belongs in RAG. Parametric knowledge, static documents, real-time data, and user-specific data each need different architectures. The PM routes knowledge type to delivery mechanism.

3. Legal RAG systems still hallucinate 17-33% on case citations. RAG reduces hallucination. It does not eliminate it. The system has to be designed for residual failure, not marketed as "hallucination-free."