Conversation Evals

The Story

Consider a cooking assistant that a user has been chatting with for several turns. The user mentions early on: “I'm now dairy-free — please adjust all suggestions.” The bot acknowledges, handles the next message fine, then two turns later recommends a cream-based risotto. The individual response is fluent, well-structured, correctly formatted. But the session failed because the bot didn't retain a live constraint stated three turns earlier.

Langfuse's multi-turn evaluation cookbook uses almost exactly this pattern — forgetting dietary restrictions — as the motivating example for why single-turn evaluation is insufficient. The MultiChallenge benchmark formalizes this as two distinct failure classes: instruction retention and inference memory.

Consider a banking support bot that begins in-domain: account balances, transaction disputes, payment schedules. By turn 6, the user asks a loosely related question about mortgage rates. The bot answers — but keeps answering mortgage questions for the next five turns, dropping the original support persona and ignoring the scope of the conversation. That's not a bad-response problem. It's a self-coherence failure.

Consider a support bot that eventually resolves a billing issue — but only after the user repeats their account number three times, rephrases their question in four different ways, and re-enters details the bot should have retained from the first message. The conversation “succeeds” in a binary sense. The experience is terrible. In customer-experience research, this pattern has a name: high effort. And high effort drives churn.

Three sessions. Three different ways to fail a conversation. None of them are visible if you only evaluate individual turns.

The Core Idea

Per-turn quality does not sum to session quality. That sentence is the entire reason this topic exists.

A bot can have excellent groundedness, relevance, and fluency on each individual turn and still fail the conversation. It fails by forgetting an earlier constraint. It fails by drifting off-domain. It fails by forcing the user to do all the repair work. None of these failures appear in a per-turn eval score. They only appear when you evaluate the session as a whole.

The shift is from evaluating responses to evaluating experiences. And the metrics that matter are fundamentally different from the single-turn triad.

Goal completion across the thread. Did the conversation accomplish what the user came for? Not “did turn 4 answer the question” but “across the full session, was the user's intent resolved?” LangSmith names this explicitly as semantic outcome — what actually happened versus what the user was trying to do.

Knowledge retention across turns. Did the system remember constraints, preferences, and facts stated earlier? The mtRAG benchmark found that state-of-the-art systems still struggle badly on later turns, unanswerable questions, and non-standalone questions — questions that only make sense given prior context.

Topic adherence. Did the system stay within its defined domain and persona? Ragas exposes this as a dedicated metric. MultiChallenge tests self-coherence explicitly.

Conversational burden — the metric most teams never measure. How much effort did the user expend to get the AI to do its job? Turns-to-resolution, number of rephrases, number of times the user repeats structured facts, escalation count. If the user needs 15 turns of rephrasing and correction to get a useful answer, the agent “succeeded” but the product failed.

What the Research Confirms

The TD-EVAL framework argues the field needs both fine-grained turn-level analysis and holistic dialogue-level comparison, precisely because either one alone misses something. Turn-level catches local policy violations, factual errors, and format breaks. Session-level catches drift, memory loss, goal failure, and burden.

The strongest case against session-level evaluation is that it's noisier, less reproducible, and harder to debug than turn-level evaluation. That position isn't wrong — it's incomplete. The direction is not “replace turn-level with session-level.” It's layer them.

Where This Hits in Production

In enterprise, “multi-turn” often means multi-session. The customer comes back tomorrow and expects the system to remember the case, the preference, or the prior investigation. A wealth management chatbot that forgets a client's risk tolerance between sessions isn't a memory bug. It's a trust violation.

Regulated conversations need conversation-level traceability. In financial services, FINRA's 2026 oversight report explicitly includes generative AI under Books and Records requirements. In healthcare, HIPAA's Security Rule requires audit controls for system activity involving protected health information. Regulated conversational AI needs reconstructible threads, retained trace context, and the ability to answer: what did the model see at each turn, what did it say, and why?

Multi-turn RAG has its own evaluation surface. Your RAG triad metrics from Topic 14 still apply, but they need to be computed per-turn AND aggregated across the session to catch failure modes that only emerge over time — retrieval instability across turns, accumulated context errors, and progressive drift of answer relevance.

Connecting the Dots

The relationship between the RAG triad and conversation eval is dimensional. The triad evaluates each turn independently. Conversation eval adds the temporal dimension: did the triad metrics stay stable across the session? Did retrieval quality degrade on later turns because the conversation history overwhelmed the context window?

On the autonomy spectrum, conversation evaluation becomes more critical as the AI handles more turns without human intervention. A Level 1–3 product barely needs session-level eval. A Level 4–5 product lives or dies on knowledge retention and coherence. A Level 6–7 product needs conversational burden as a core metric — because the agent is supposed to reduce user effort, not increase it.

The moat is in the conversation data. Your multi-turn evaluation dataset — full transcripts of real user sessions, with annotated failure points and burden markers — encodes how YOUR users actually interact with YOUR product. A competitor can replicate your single-turn eval in a day. They can't replicate 10,000 annotated multi-turn sessions.

In Practice: Building Multi-Turn Eval Infrastructure

LangSmith handles multi-turn through threads and message histories. Its three evaluation dimensions: semantic intent (what the user was trying to do), semantic outcome (what actually happened), and trajectory (how the conversation unfolded, including tool-call sequences).

Ragas provides explicit multi-turn structures: MultiTurnSample and EvaluationDataset. Its metric surface includes Topic Adherence, Tool Call Accuracy, Tool Call F1, and Agent Goal Accuracy.

Langfuse provides the most practical cookbook: find bad traces, save the full conversation history up to the point of failure, and use that as a regression test. That pattern — production failures becoming evaluation cases — is the fastest path to a useful multi-turn eval set.

The bridge to Topic 16 is direct: conversational drift is the conversational analogue of agent plan drift. Topic 15 teaches you to see the conversation as a path. Topic 16 extends that logic into explicit agent trajectory evaluation with tool selection, handoff fidelity, and error recovery.