Non-Determinism

The Story

Two weeks after a cloud infrastructure company shipped an AI ticket-triage system, enterprise customers started complaining about queue ping-pong: the same intermittent-storage-latency ticket got routed to network Monday, storage Tuesday, and performance engineering Wednesday. Severity flipped between Sev-2 and Sev-3 on the same ticket depending on when the system processed it.

The work used to take a junior engineer ninety seconds per ticket. The AI did it in two. The team had tested it thoroughly — same prompt, same model, same schema, same output format. The demo worked perfectly. They shipped.

The system was not broken. It was unstable. Same input, same model, meaningfully different routing decisions across runs. The team validated accuracy once and saw it work. Nobody measured whether the decisions stayed consistent when you ran the same case twenty times.

The pattern is common enough that the FINOS AI Governance Framework — used by financial institutions to assess AI systems — explicitly flags it: non-deterministic behavior can cause varying risk assessments, inconsistent customer responses, or unreliable compliance checks from identical inputs. FINOS named it because its authors had watched it ship and fail in regulated environments more times than they would like.

The Core Idea

Traditional software gives you a guarantee that most product teams take for granted: determinism. The same input produces the same output. If the search for "red shoes" returns Nike Air Max today, it returns Nike Air Max tomorrow. If the result is wrong, it's wrong the same way every time. You find the bug, fix it once, and it stays fixed.

LLMs don't work this way. When a model generates text, it's sampling from a probability distribution over possible next words. Temperature and top-p introduce controlled randomness into every response. Even with seed parameters and matching system fingerprints, outputs are described as only "mostly identical."

Independent research puts numbers on this. Research has documented accuracy swings of up to 15 percentage points across naturally occurring runs of the same eval, with best-versus-worst gaps reaching 70 points on certain tasks. Hardware differences, GPU count, and even evaluation batch size can materially change outputs — even under greedy decoding.

Non-determinism breaks three assumptions that product teams carry without realizing it.

You can't test once and trust the result. In traditional software, one pass through your test suite tells you whether the code works. With LLMs, one pass tells you what happened this time. Anthropic's evaluation guidance recommends running multiple trials per test case specifically because outputs vary between runs. A single trial is one sample from a distribution. You need the distribution itself.

"It worked in the demo" means nothing about production. Every product team has experienced this. The demo to leadership was flawless. The CEO saw exactly the output they wanted. Then production quality was noticeably worse — not because anything changed, but because the demo was one sample from a distribution, and it happened to land near the best case. Production shows the mean.

Task completion is not task success. This is the Completion Fallacy, and it's the most underestimated failure mode in AI products. The support ticket was routed — but to the wrong team. The email was sent — but it hallucinated the recipient's name. The report was generated — but it pulled data from the wrong quarter. From a monitoring dashboard, every one of these looks like a success. The API returned 200. The task reached its end state. The system "worked."

What this means for how you think about quality: quality in AI products isn't a number. It's a distribution. You don't have "accuracy = 85%." You have "accuracy ranges from 72% to 94% across runs, with a mean of 85% and a standard deviation of 6%." That variance matters enormously. In regulated industries, a system that's right 85% on average but occasionally drops to 72% is potentially dangerous. The mean looks acceptable. The tail is where the damage lives.

Where This Hits in Production

Auditability in regulated industries. When a financial services regulator asks "why did the system approve this transaction on Tuesday but deny the same profile on Thursday?" — non-determinism turns that from a bug investigation into an existential question. Regulated teams must log the model's system fingerprint, prompt version, retrieval inputs, and full output as audit artifacts.

Multi-tenant context window variance. In enterprise B2B platforms, the same model serves different customers with different retrieved documents. Customer A's query might fit comfortably in the context window. Customer B's query, with larger retrieved documents, might overflow the context window, silently truncating critical system instructions. Same model. Same prompt template. Different behavior.

Cost unpredictability. Non-deterministic agents can consume tokens at wildly different rates. An agent that takes 2 efficient steps for one query might take 15 looping, hallucinated steps for a nearly identical query. For flat-rate enterprise SaaS pricing, this destroys margin predictability.

Connecting the Dots

Here's the practitioner insight that most eval tutorials miss: non-determinism isn't a problem to eliminate. It's a property to manage. And the first management decision is deceptively simple: evaluate at the temperature you ship at. If you eval at temperature 0 but ship at temperature 0.7, your eval results are meaningless.

On the AI autonomy spectrum, this distinction matters enormously. At Levels 1-3 (autocomplete, chatbots, assistants), non-determinism produces slightly different phrasings of the same recommendation. At Levels 5-7 (agents, autonomous systems), non-determinism means the agent might take a completely different sequence of actions on different runs. The email gets sent to a different recipient. The database query targets a different table.

Mature teams don't fight for byte-identical outputs — they define invariants (the things that must stay the same) and tolerances (the things that can vary).

The routing decision must be stable. The phrasing can vary. The escalation flag must be consistent. The word choice doesn't matter.

Remember This

1. The same prompt produces different outputs on different runs. Temperature zero doesn't fix this. GPU hardware, batch size, and serving conditions inject variance that even the model provider says they can't eliminate.

2. "Task completed" is the most dangerous metric in AI — the Completion Fallacy. The JSON parsed, the API returned 200, the task reached its end state. And the routing decision was wrong.

3. Quality is a distribution, not a number. The mean looks acceptable. The variance is where the production failures live. Measure both.

In Practice: The Stability Harness

Here's what the support-ticket team should have built before trusting the routing system in production. Run each test case from your golden dataset not once, but 20 times. Same input, same model, same prompt. Log every output.