- 01. Why CI/CD eval gates and production monitoring are different tools solving different problems — and why confusing them leaves you blind.
- 02. The three ways production monitoring fails in practice: silent decay, over-triggering guardrails, and alert fatigue.
- 03. How the best teams combine sampling strategy, online evaluation, and dataset versioning to learn from live traffic.
The Story
Consider a legal research assistant that passed its CI eval gate cleanly every sprint. The team had a well-maintained dataset of 90 representative cases. The gate ran on every merge. Quality looked stable. Then a new regulatory filing category became common in user requests — a document type that had rarely appeared when the dataset was built. The assistant handled it poorly: it retrieved technically relevant passages but missed the specific clause structure that legal professionals actually needed.
Accuracy on the old test cases stayed high. Accuracy on the new request pattern was bad. Nobody knew because the old test cases still passed. This is the gap production monitoring exists to fill.
CI/CD evals validate whether the system still passes your known tests. Production monitoring watches whether users are being served well on traffic that didn't exist when you wrote the tests. OpenAI's evaluation best practices documentation is direct about this: teams should continuously evaluate, monitor for new nondeterministic cases, and grow the eval set over time. The mechanism for discovering those new cases is production monitoring — not CI.
Now consider a consumer support assistant with a topic-relevance guardrail — an automated check that flags responses about topics outside the assistant's intended scope. The guardrail is well-intentioned. Over time, the LLM judge running the guardrail starts flagging routine answers about product features as "possibly financial advice" when users phrase questions in a certain way. The flagging rate climbs from 2% to 11%. Legitimate responses start getting blocked.
Infrastructure metrics show nothing. The server is healthy. Latency is fine. Cost is stable. The product is silently breaking for a growing subset of users, visible only through the guardrail's false-positive rate — which nobody set up an alert on because when it launched at 2%, it felt like background noise.
Then there's alert fatigue. A platform team that had built a serious monitoring system: online evaluators for safety, relevance, and groundedness; dashboards with cost, latency, and token usage; feedback-score tracking; alerts for rate-limit violations, latency spikes, and quality dips. Three months later, alerts fired multiple times per day across all categories. Engineers started ignoring notifications. A significant safety regression went unnoticed for four days because it appeared alongside 30 other alerts that had been generating low-urgency pages all week.
Three systems, three different ways production monitoring failed. None of them involved a broken server.
The Core Idea
The difference between CI/CD evaluation and production monitoring is the question each one answers.
CI/CD evaluation asks: does the system still pass the cases we know about? Production monitoring asks: is the system serving users well on the cases we haven't seen before?
Both questions matter. Neither one makes the other unnecessary. Teams that run only CI evals are protected against known regressions and blind to live drift. Teams that run only production monitoring are watching the current state but have no guard against introducing regressions through prompt or model changes. The two layers are complements — the same "Swiss cheese" logic that applies to the eval stack overall.
Production quality monitoring runs
automated evaluations on live traffic.
Does the system still pass the cases we know about?
Is the system serving users well on cases we haven't seen before?
Test-driving the car tells you if it still handles correctly. You need both.
Three things production monitoring must do that CI/CD evaluation cannot.
Watch live traffic for new failure patterns. Test datasets are necessarily historical. Production traffic is live. A new document type, a new user phrasing pattern, a new edge case that users discover through organic use — none of these appear in the golden set until someone manually promotes them. Production monitoring surfaces these as anomalies in quality scores, negative feedback spikes, or unusual guardrail firing rates.
Detect guardrail and evaluator drift over time. Automated judges aren't static. A topic-relevance judge that worked well at launch may become too aggressive or too permissive as the underlying model updates or as the distribution of user inputs shifts. If you're not monitoring the judge's own output distribution — false-positive rate, firing frequency, score spread — you won't know it's drifting until users complain loudly enough.
Feed the dataset versioning loop. This is the mechanism that closes the gap between production monitoring and CI/CD evaluation. Production monitoring finds new failure patterns, annotation queues surface them to reviewers, confirmed failures get promoted into the eval dataset, and the CI gate now catches that failure class going forward. The teams that build durable eval infrastructure treat their dataset as a living system that learns from production.
Where This Hits in Production
Online evaluation in LangSmith targets real traffic, not curated datasets. LangSmith's online evaluators run against actual runs and conversation threads from production tracing. They evaluate individual turns and — critically — whole conversation threads using metrics like semantic intent, semantic outcome, and trajectory. That last metric connects production monitoring directly to session-level quality concerns.
Datadog now treats quality as a first-class monitoring surface. Its LLM Observability stack monitors, troubleshoots, and evaluates LLM applications for quality, privacy, and safety. Teams can run custom LLM-as-a-judge evaluations, use Datadog-managed evaluation templates (for prompt injection, hallucination, topic relevancy, toxicity, and failure-to-answer), or submit external evaluation results. Those scores feed directly into dashboards and monitors — meaning a groundedness regression has the same alerting infrastructure as a latency spike.
Sampling strategy is where cost meets coverage. Running LLM-as-a-judge evaluation on 100% of production traffic is expensive. Running it on 1% misses rare-but-important failure modes. The practical solution: send 10% of all traces to an annotation queue for spot checks while sending 100% of negatively-rated traces regardless of sampling. That combination gives broad coverage on routine traffic and full capture on high-signal events without paying judge costs on every request.
Every incident investigated.
Prevents slow drift from compounding.
Connecting the Dots
Production monitoring is where the full eval stack starts to compound. CI/CD gates stop regressions from entering production. Production monitoring catches what changes after production — new traffic patterns, model drift, guardrail erosion. Trace debugging is what you do when production monitoring surfaces an anomaly: the monitoring signal tells you that quality dropped on a specific request category; the trace tells you which span produced the degradation. Human review resolves ambiguous cases that monitoring surfaces but cannot definitively label.
The connection to dataset quality is the most underappreciated one. Most teams think of their eval dataset as a static asset they maintain by occasional addition. Every failure pattern that production monitoring surfaces and a reviewer confirms is a new row in the dataset and a new CI gate. Over time, the CI gate covers more of the input space. Over time, the eval suite gets harder to fool with a regression.
The Pass^k reliability multiplier from Topic 16 shapes what production monitoring must track for agents. An end-to-end quality drop from 85% to 72% tells you something degraded — but not what. If the agent chains 8 steps, any one of those steps could have slipped from 98% to 96% per-step reliability, and that's enough to cause the observed drop. Production monitoring for agentic systems needs per-step reliability metrics, not just an aggregate score. When the end-to-end number moves, per-step tracking is how you diagnose which step's reliability degraded and by how much — without it, you're staring at a symptom with no way to locate the cause.
This is also where compliance pressure becomes real for regulated industries. HIPAA's technical safeguards require mechanisms to record and examine activity in systems containing electronic protected health information. The EU AI Act requires high-risk AI systems to maintain automatic logging capabilities over the system's lifetime. FINRA's 2026 report elevates generative AI and books-and-records obligations as a supervision priority. In regulated settings, production monitoring isn't just a quality practice — it's part of the compliance architecture.
Common Mistake
Treating production monitoring as a cheaper version of offline evaluation.
The question production monitoring answers is fundamentally different from the question offline evaluation answers. Offline evaluation tells you whether known tests still pass. Production monitoring tells you whether users are being served well on live traffic.
If you run offline evals infrequently and call that "monitoring," you miss live drift. If you run production monitors and call that "sufficient for CI," you have no guard against introducing regressions through prompt or model changes.
A system can be fast, cheap, and functionally healthy while producing wrong answers on a growing share of user queries.In Practice
Design the alert hierarchy before you wire up the monitors. Three tiers: Critical (safety violations, security failures) pages on-call immediately with no sampling. Warning (quality score drift, guardrail false-positive rate exceeding baseline, significant feedback score drop) opens a ticket, reviewed within 24 hours. Informational (cost or latency threshold brushes, minor score fluctuations) logged to dashboard, reviewed weekly. Without this hierarchy, every signal fires at the same urgency and none of them get acted on.
Set up the sampling strategy to balance cost and coverage. Random sampling at 10-20% of traffic for broad quality coverage. Risk-weighted routing at 100% of negatively-rated, flagged, or high-cost traces for high-signal capture. LangSmith's automation rules or Datadog's span filters handle this routing at configuration time, not at instrumentation time.
Build the production-to-dataset feedback loop from day one. Enable annotation queues. Route confirmed failure traces to the eval dataset (versioned, dated). Pin CI experiments to the dataset version that was current when the model shipped. Review and promote new production failures monthly. The teams that do this have eval suites that get better with time. The teams that don't have eval suites that slowly become less representative of the product they're actually shipping.
The bottom line: CI/CD evals test what you know. Production monitoring watches for what you don't know yet. You need both. The dataset versioning loop is the compound interest: production monitoring finds new failures, reviewers confirm, dataset grows, CI gate gets stronger. Skip the loop and the two layers never learn from each other.
Per-tenant and per-segment monitoring isn't optional in enterprise systems. A global average score that looks stable can hide concentrated failures in one customer, one model route, or one workflow type. Segment the dashboards. Segment the alerts.