Multi-Turn State

The Story

A workforce management SaaS company built an AI assistant for HR directors. Typical sessions ran 15-25 turns as directors worked through complex decisions: "Show me attrition by department... now compare that to industry benchmarks... what if we increase the engineering budget by 15%... actually, let's look at the sales team instead..."

For the first 12 turns, performance was strong. At turn 18, the assistant started repeating information. At turn 22, it confused the engineering headcount scenario with the sales team analysis from turn 15. At turn 25, it generated a recommendation that directly contradicted its analysis from turn 8 -- and didn't notice the contradiction.

The HR directors described it consistently: "It's like the assistant has early-onset amnesia. It's sharp for the first 15 minutes, then it gets confused." The window wasn't full. It was polluted.

The Core Idea

The first design insight: not all state is the same kind. Production systems manage three distinct state types, each with different preservation rules.

Conversational state -- what was said. The literal exchange. This is the most voluminous and the most compressible. The user's exact phrasing of turn 3 rarely matters at turn 20.

Task state -- what's been accomplished. Decisions made, analyses produced, recommendations generated. Task state IS the work product. Losing task state is losing the work.

Compliance state -- what was agreed to. In enterprise products, this tracks what was consented to, disclosed, approved, and committed to. Compliance state gets the strictest preservation policy: verbatim, never compressed, persisted to an external audit log.

The dominant architecture for multi-turn state management is the sliding window + summarization hybrid. The context window is partitioned into zones: an active window (last 3-5 turns, verbatim), a summary zone (compressed earlier turns), pinned state (critical items that persist regardless), and dropped turns (noise that's been removed).

A 25-turn conversation managed this way uses ~6,000 tokens of history instead of ~42,000. Quality stays at 89% instead of degrading to 68%.

Where This Hits in Production

The 50-turn cliff is real. Without state management, a 50-turn conversation will have consumed approximately 60,000-80,000 tokens of history alone. Even models with 200K context windows exhibit measurable quality degradation at this volume. State management isn't an optimization -- it's a requirement for any product where sessions exceed 15-20 turns.

State management failure looks like model failure. When the HR director says "the assistant gets confused after 15 minutes," nobody suspects state management. They suspect the model. The PM must instrument state management so that when quality degrades at turn 20+, the debugging trail points to the state, not the model.

Remember This

1. Multi-turn conversations require three distinct state types: conversational (what was said), task (what was accomplished), and compliance (what was agreed to). Each has different preservation rules. Losing task state kills coherence. Losing compliance state creates legal exposure.

2. The sliding window + summarization hybrid is the dominant production pattern: last 3-5 turns verbatim, earlier turns summarized with priority weighting, critical decisions pinned regardless of age. A 50-turn conversation managed this way uses ~8,000 tokens instead of ~70,000.

3. State management failure looks like model failure. When quality degrades at turn 20+, the debugging trail should point to state first.