A/B Testing for AI

The Story

A travel booking platform redesigns its AI search assistant. The new version uses a better retrieval pipeline and a more recent model. Offline evals are convincing: relevance scores improve by 12%, hallucination rate drops by a third, and format compliance is near-perfect. The team ships the new assistant to all users on a Tuesday.

By Friday, bookings are down 4%. The new assistant produces more detailed, more accurate results -- and they take 800ms longer to render. Users on mobile, who represent 60% of traffic, see the loading spinner just long enough to lose patience and switch to the manual search filters. The offline eval measured output quality. It never measured what that quality costs the user in wait time.

LaunchDarkly's experimentation documentation makes the production question explicit: experiments measure the effect of feature variations on end-user metrics including page views, clicks, load time, and infrastructure costs. That's not a richer version of the offline eval -- it's a fundamentally different measurement. The offline eval asks "is the output better?" The A/B experiment asks "does the change make the product better for users, measured by what users actually do?"

A second team takes the careful approach. They're replacing their customer support agent's underlying model with a newer, cheaper version. Before any user sees the new model's responses, they run it in shadow mode: every incoming support request gets processed by both the old and new model. The old model serves the actual response. The new model's output goes to logs, not to users. After two weeks and 50,000 real requests, the shadow system hasn't crashed, latency is acceptable, tool-calling behavior is stable. The team promotes the new model.

Within a week, customer satisfaction scores drop. The shadow deployment proved the new model could handle production traffic technically. It never proved users would prefer the new model's actual responses. The team validated survivability and confused it with quality.

A third team runs a proper A/B experiment. They split traffic 50/50 between two prompt variants. After ten days, Variant B shows a 3.2% improvement in click-through rate with a p-value of 0.04. They ship Variant B. A month later, a data scientist discovers a sample ratio mismatch -- the actual traffic split was 53/47. The SDK was caching flag evaluations on one path, causing a systematic imbalance. The 3.2% improvement might have been real. It might have been an artifact of the asymmetric allocation.

Three teams, three different methods, three versions of the same mistake: using the right production validation technique to answer the wrong question, or using the right technique with broken plumbing.

The Core Idea

Offline evals and production validation answer different questions, and no amount of sophistication in one substitutes for the other.

An offline eval tells you whether a change looks better on a curated or sampled dataset. It controls the inputs, applies consistent scoring, and produces reproducible results. That's its strength: scientific clarity under controlled conditions. That's also its limitation: it cannot capture latency under real load, user behavior in response to changed outputs, business metric impact, session-level effects over multiple interactions, or failure modes triggered by the long tail of production traffic that no golden dataset fully represents.

Production validation methods fill that gap -- but they're not interchangeable. Each answers a specific question, and using the wrong one wastes time or, worse, produces false confidence.

Shadow deployment answers: "Can the new system survive production traffic?" The candidate system receives copies of real production requests and processes them, but its responses are discarded -- the stable system still serves users. You learn whether it crashes, whether its latency is acceptable, whether its tool calls are well-formed. What you don't learn: whether users would prefer the shadow system's outputs. Shadowing is a technical validation method, not a product validation method.

Canary analysis answers: "Does the new system hold up as real exposure increases?" A small percentage of actual users receive the candidate system's responses. Unlike a full A/B experiment, the goal isn't causal impact measurement with statistical rigor. The goal is progressive exposure with automated rollback: increase the canary weight in stages, monitor key metrics at each stage, and abort if the metrics regress. Canary analysis is less rigorous than A/B testing but safer than full rollout.

A/B experiment answers: "Did this change cause a measurable improvement in the outcome we care about?" This is the only method that establishes causation. Users are randomly assigned to variants. The assignment is persistent. Metrics are tracked per variant over a sufficient time window with appropriate statistical controls -- SRM detection, CUPED variance reduction, multiple comparisons correction, and sequential testing.

Where This Hits in Production

The sequence is: offline, then shadow, then canary, then A/B. Not a menu -- a ladder. Each stage answers a question that gates the next. Skipping a stage means accepting the risk of the question that stage would have answered. Sometimes that risk is acceptable -- a trivial prompt tweak might go straight from offline eval to canary. But the team should be making that skip consciously, not by default.

The randomization unit determines what your experiment can measure. Randomize by user in a B2B product where five users share one organization account, and you've contaminated the experiment. Two users in the same account see different versions. They compare notes. One reports a bug that's actually a feature the other user has. For enterprise and B2B products, the default should be account-level or organization-level randomization.

Shadow deployments have a data governance surface. Because live production requests are copied to the mirrored service, any sensitive data in those requests reaches the shadow system. In regulated environments -- healthcare, finance, legal -- the shadow path must obey the same privacy, retention, and access controls as primary production.

Feature flags connect experimentation to observability. Datadog's feature-flag tracking attaches flag evaluation data to Real User Monitoring telemetry. That means you can see which users saw which variant and whether the variant correlated with performance regressions, errors, or experience degradation. When something goes wrong, you can filter by variant to determine whether the experiment caused it.

The Traps

Trap 1: Invalid experiment infrastructure. The experiment runs, produces a confident result, and the team ships the winner. But the experiment was broken from the start. LaunchDarkly's health checks look for five specific failures: exposure mismatches, SRM, randomization-unit conflicts, missing metric events, and targeting-rule mismatches. An experiment with any of these problems is not producing valid results. It's producing confident noise.

The fix isn't statistical sophistication -- it's plumbing hygiene. Before interpreting any result, check the health indicators. Before running any experiment, validate that the randomization unit matches your product's natural boundary, that the SDK is evaluating flags correctly, and that metric events are flowing from all variants equally.

Trap 2: Offline eval as permanent substitute. The team has a strong offline eval suite. It's well-curated, regularly updated, and includes production traces. They use it as the sole validation method for every change. This works until a change affects something the offline eval doesn't measure: latency perception, session-level behavior, user trust, or the interaction between two changes that were evaluated independently. Offline evals are the wind tunnel. They're essential and insufficient. The road test still happens.

Remember This

Three methods, three questions. Shadow deployment: can the system survive real traffic? Canary analysis: does it hold up as exposure increases? A/B experiment: did the change improve outcomes for users? These are a sequence -- offline, shadow, canary, A/B -- not a menu where you pick one.

The randomization unit is the most important design decision in production experimentation. Get it wrong and the experiment is invalid before it starts. For B2B products, randomize by organization, not by user.

An experiment with a confident result and broken plumbing is worse than no experiment. Check the health indicators -- SRM, exposure balance, metric completeness -- before interpreting any result.

Offline evals are the wind tunnel. Production experiments are the road test. The wind tunnel is essential. The car still has to drive in the rain.