Memory Architecture

The Story

Three weeks after a wealth-management platform shipped its AI advisor to 500 consultants, the same complaint arrived from every region: "the AI doesn't remember us." The model itself had zero persistent state — and the team had never designed any. The AI was excellent at one-off tax, allocation, and market questions. It was a stranger to every client, every morning.

Three weeks in, the feedback is consistent and frustrated.

A consultant spends 20 minutes discussing a client's retirement timeline, risk tolerance, and inheritance goals. The next day, she opens a new session to continue. "Based on what we discussed yesterday about the Patel account..." The AI responds as if they have never spoken. Every piece of context is gone. She re-explains everything. Trust drops a notch.

A different consultant notices the opposite problem. The AI keeps referencing a previous client's preferences in a new client's session. "As you mentioned, you prefer conservative allocations" -- except this client never said that. The previous client did. The memory system is leaking context between sessions. Compliance pings the team.

A third consultant discovers that after a long session covering multiple clients, the AI confuses details between them. It attributes one client's tax situation to another client's portfolio. The current session's working memory is overwhelmed -- too many facts, insufficient structure. The AI sounds confident and gets the wrong account's number.

Three failures. Three different kinds of memory gone wrong. Missing episodic memory (yesterday's conversation is lost). Leaking semantic memory (one user's data contaminating another's). Overloaded working memory (too much in the current session). Each one needs a different architectural fix. None of them is the model's fault.

The Core Idea

The single most important fact about AI memory: the model has none. Not limited memory. Not short-term memory. Zero. Every call to the model processes the context window from scratch and generates a response. The model does not remember anything between calls. It does not forget -- it never stored anything in the first place.

What users experience as AI memory is entirely manufactured by the harness -- the system wrapping the model. The harness stores information persistently and loads the relevant pieces back into the context window before every inference call. The model processes what it sees and answers. The illusion of continuity is a product of context engineering. Not model capability.

This is the Tracks layer in the seven-letter framework from The CONTEXT Stack -- the T in CONTEXT. It is the layer most often confused with the kNowledge layer from Knowledge at Inference. The distinction matters: kNowledge is shared world facts (the company's return policy, regulatory requirements) -- the same for every user. Tracks is user-specific, session-specific -- what this user said five minutes ago, what this user prefers based on last month's interactions. Same building. Different functions. Different architectures.

Think of it like a hotel concierge who sees hundreds of guests daily. They don't remember you intrinsically. But your preferences are in the guest file (semantic memory), your check-in conversation is in today's log (short-term memory), your previous stays are in the CRM (episodic memory), and the notes in front of them right now are their working memory. The guest experiences a concierge who "remembers everything." The reality is a well-organized information system.

Working memory is what's in the context window right now. It's the most transient type: exists only for this inference call. Free but limited by window size.

Short-term memory is the conversation history within the current session. As the conversation grows, it consumes more tokens and eventually requires management: summarisation, selective loading, sliding windows. A 50-turn conversation can consume the entire context window if left unmanaged -- which is exactly the failure mode catalogued in Context Rot.

Episodic memory is what happened in past sessions. "Last Tuesday you decided to go with Vendor B." It requires persistent storage and a retrieval mechanism.

Semantic memory is long-term user-specific facts extracted from episodes. "This user prefers conservative allocations." A fact takes fewer tokens than a full conversation transcript and changes more slowly than episodes accumulate.

The hybrid memory structure that works in production uses extracted entities (always present, ~200-500 tokens), a conversation summary of older turns (~300-800 tokens), the last 3-5 turns verbatim (~1,000-3,000 tokens), and retrieved episodic context only when relevant (~500-1,500 tokens). Total: roughly 2,000-6,000 tokens per query -- a fraction of most context windows -- while preserving the information the model needs most.

Where This Hits in Production

Privacy and compliance. Memory systems store user data. Under GDPR, CCPA, and the dozens of similar laws now in force, users must be able to view, edit, delete, pause, export, clear what the AI remembers about them -- as first-class features, not afterthoughts. ChatGPT's memory feature ships keyword search, sort-by-recency, restore-prior-versions, and delete-specific-memories, setting the user expectation for every AI product that follows.

Cross-session consistency. When a user returns after a week, the AI should reference relevant past context without being creepy ("I remember you mentioned your daughter's wedding!") or amnesic ("I have no record of our previous conversations"). This is a product judgement, not a technical default -- and it is the line between a memory system that compounds trust and one that quietly creates discomfort.

Multi-tenant memory isolation. In B2B platforms, memory leakage between tenants is a data breach. In multi-user systems within the same tenant, leakage between users is equally dangerous. The architecture has to enforce isolation at both tenant and user levels with audit logging that proves compliance. The wealth-management story above is not just a quality bug -- in a regulated context, it is a reportable incident.

Memory tokens aren't free. Loading 3,000 tokens of memory per query, across 1 million queries per day, at $3 per million input tokens = $9,000/day in memory cost alone.

Connecting the Dots

Memory is the CONTEXT layer that sits at the boundary between Level 1 (context engineering) and Level 2 (harness architecture). The memory design is context engineering -- what types, what structure, what enters the window. The memory system is harness architecture -- how it is stored, retrieved, and managed across days and weeks. This is why memory leads directly into the 3C lifecycle and multi-turn state work in the next layer of the series.

The eval connection. A product that remembers user preferences should be evaluated on questions standard output-quality evals miss entirely: does it consistently apply those preferences? Does it reference the right past conversation? Does it AVOID referencing the wrong one? Those are memory-specific eval dimensions -- the discipline of conversation evals from the AI Evals series.

On the autonomy spectrum. At low autonomy levels, the harness manages memory -- the PM designs what gets stored, the system handles the lifecycle. At high autonomy levels, the agent manages its own memory, deciding what to remember and when to forget. Anthropic's Claude Code spawns sub-agents with independent context windows rather than summarising accumulated context -- the model creates fresh working memory rather than compressing old, echoing the reset beats compress finding from Context Rot.

In Practice

Here's how a production team designs memory for a customer success AI. The key: the memory system isn't just storing and retrieving -- it's managing a lifecycle. Facts enter, get used, age, and eventually leave.

Remember This

1. The model has zero persistent state. AI memory is entirely created by the harness -- storing information persistently and loading the relevant pieces into the context window before each call. The illusion is a product of engineering, not capability. Build the illusion deliberately or it will fail unpredictably.

2. Four types of memory serve different purposes: working (current context), short-term (this session's history), episodic (past sessions' specific interactions), semantic (extracted long-term facts). Confusing them produces four different failure modes -- amnesia, leakage, overload, drift.

3. Memory without decay causes context rot. Design for forgetting: recent interactions weigh more, semantic facts persist, episodic details age out. And give users control. They must be able to see, edit, and delete what the AI remembers -- both because the law requires it and because the trust depends on it.