How to Measure AI Accuracy in Production
An AI system's accuracy on the day it ships is the highest accuracy it will ever have.
Not because the model gets worse. The model is deterministic-ish under load. The world it operates in is not. Users ask questions the test set did not anticipate. Vendors update models silently. Retrieval indexes go stale. Prompts get tweaked. Business rules change. Each of these is a small drift. The compound effect is a system that scored 0.94 on offline evaluation and produces 0.71-quality outputs three months later, with nobody quite sure when the erosion started.
Production accuracy measurement is the discipline that catches this. It is different from offline evaluation, and most teams underinvest in it because it looks like the same problem. It is not.
The Difference Between Offline and Production Measurement
Offline evaluation runs against a fixed test set. Repeatable. Deterministic. Used to answer the question "is this change safe to ship." Every model swap, every prompt edit, every retrieval config change gets run through the offline suite before deploy.
Production measurement runs against live traffic. Not repeatable in the same sense. Not deterministic. Used to answer the question "is the world still the shape our system was designed for." It runs continuously after deploy, not just before.
The two jobs use overlapping metrics but different infrastructure. Offline evaluation has a test set. Production measurement has a stream. Offline evaluation can afford expensive checks because it runs once per deploy. Production measurement has to be cheap because it runs against every request or a sample of every request.
The teams that ship reliable AI systems have both. The teams that skip production measurement have the illusion of quality that lasts until users start complaining.
The Four Ways to Measure Accuracy Against Live Traffic
Every mature production measurement stack combines some or all of the following.
Human-labeled sampling. Pick a sample of production traffic, have humans label whether the outputs were correct, aggregate the labels into a running accuracy metric. This is the most trustworthy signal and the most expensive to produce. Sample size drives cost. A team that labels 100 samples per day gets a slow-moving accuracy number. A team that labels 10,000 per day gets a fast-moving one at 100x the cost.
Reference-free automated evaluation. Use LLM-as-judge to score outputs against criteria that do not require a ground-truth answer. Faithfulness, groundedness, answer relevance. Runs against every production request or a large sample without human involvement. Cheaper than human labeling. Correlates around 81 percent with human scores when the judge prompt is well-designed. Not as trustworthy as human labels, but scales to volumes human labeling cannot touch.
User signal capture. The user's own behavior is a signal about accuracy. Retry rates, escalation rates, thumbs-up or thumbs-down feedback, session abandonment, whether the user pasted the output somewhere or immediately rewrote it. Every user action after receiving an AI output is evidence about whether the output worked. Capturing these signals is free at the technical level and requires product instrumentation to be useful.
Drift detection. Statistical monitoring of the distribution of inputs and outputs. When the input distribution shifts, the model may be operating in unfamiliar territory even if it does not know it. When the output distribution shifts, something in the pipeline changed. Drift detection does not measure accuracy directly. It measures the conditions under which accuracy is likely to have moved. Every production monitoring platform ships some form of it.
None of these are sufficient on their own. The teams that ship well combine all four. Human labels for the trustworthy baseline. Reference-free automated evaluation for scale coverage. User signals for the free ground truth. Drift detection for the early warning.
Sampling Strategy
Production accuracy measurement rarely runs against every request. The cost and the latency add up. Sampling is the practical shape.
The sampling strategy is more important than most teams realize. Sampling uniformly at random gives you a representative average, but hides accuracy differences across user segments, request types, and time windows.
Stratified sampling. Divide production traffic into meaningful segments (by user tier, request category, time of day, geographic region). Sample separately from each segment. This lets you see whether accuracy is uniform across segments or concentrated failure in a specific subgroup. A system with 90 percent accuracy overall but 60 percent accuracy on a specific user segment is failing that segment while the top-line metric hides it.
Weighted sampling. Sample higher-value requests more heavily. If enterprise customers pay 10x more than free users, weighting their requests 10x higher in the sample matches the sample to the revenue distribution.
Adaptive sampling. Increase the sample rate when drift signals fire. Under normal operation, sample at 2 percent. When input distribution drift triggers, jump to 20 percent for that segment for 24 hours. This concentrates measurement resources where they are needed.
Uniform random sampling at 1-5 percent is the reasonable starting default. Every team that scales production accuracy measurement past the first six months moves to stratified or adaptive strategies because uniform sampling stops answering the questions the team cares about.
What LLM-as-Judge Can and Cannot Do at Production Scale
Automated evaluation is the practical answer to "how do we measure accuracy without paying for human labels on everything."
The 2026 state of the art. Judge models correlate about 81 percent with human labels on well-designed rubrics. This is good enough to be the default scoring layer for production sampling. Cost is roughly the cost of one additional LLM call per sampled request. Latency is one extra API round trip, or zero if the scoring runs asynchronously after the response returns.
Where LLM-as-judge falls short in production.
Judge drift. Vendor model updates change judge behavior. A system scoring 0.87 on faithfulness this month can score 0.83 next month with no change to the underlying application, purely because the judge model got updated. Log the judge model version. Never treat cross-version scores as directly comparable.
Prompt sensitivity. Different rubric prompts produce different scores on identical outputs. Two teams evaluating the same production traffic with different rubric wording can disagree by 15 points. Version the rubric. Retire old rubrics rather than updating them in place.
Domain-specific gaps. Judge models handle general-purpose evaluation well and specialized domains badly. Legal reasoning, medical accuracy, code correctness on rare languages. In these areas the judge's own knowledge is the limiting factor. Custom-trained judges or human review are the fallback.
Cost at scale. Every sampled request adds an LLM call. At 1 million requests per day and 10 percent sampling, that is 100,000 additional LLM calls per day. Even at low unit cost, that budget adds up. Platforms like Galileo Luna-2 emphasize the cost-per-eval dimension for a reason.
The User-Signal Layer Most Teams Under-Instrument
Production users tell you when the AI is wrong. Most teams do not listen.
Explicit feedback. Thumbs up, thumbs down, star ratings, "was this answer helpful" prompts. The most direct signal. Coverage rate is low (5-15 percent of users actually respond) but the signal is unambiguous when it comes in.
Implicit feedback. Retry rate on the same query. Time to next user action. Session length. Whether the user pasted the answer somewhere or immediately edited it. These signals are noisier but cover 100 percent of traffic without asking anything of the user.
Escalation rate. The rate at which the AI system handed off to a human. For a support agent, the escalation rate is the inverse of task completion. High escalation rate on a segment means the AI is failing that segment.
Return rate. Users who came back to ask the same question again in a different session. This is one of the strongest signals that the previous answer was insufficient. Detecting it requires cross-session user identification, which not every product has.
Product instrumentation to capture these signals is a small investment that produces the most trustworthy accuracy signal available. Users know when the AI is wrong faster and more accurately than any evaluator.
Drift Detection Is the Early Warning
Drift detection does not measure accuracy directly. It measures whether the conditions the system was designed for have shifted enough that accuracy is likely to have moved.
Input distribution drift. The categories of questions users are asking. The average query length. The rate of adversarial or jailbreak-style inputs. When these shift, the model is operating on a different problem than the one the offline evaluation covered.
Output distribution drift. The average response length. The rate at which the model refuses to answer. The frequency of specific tool calls. When these shift, something in the pipeline changed (prompt, model, retrieval, tool availability) even if nobody deployed a change.
Metric drift. The evaluation scores themselves moving over time. When faithfulness drops from 0.91 to 0.85 over two weeks, something is degrading. Drift detection on the scores catches the degradation before users complain loudly enough to be heard.
Drift is a leading indicator. Actual accuracy measurement is a lagging indicator. Both are needed. Drift alerts trigger deeper investigation. Accuracy measurement confirms whether the drift produced real quality loss.
What Any of This Still Cannot Tell You
A production accuracy measurement stack answers the question "was the output faithful to the input and the retrieved context." It does not answer the question "was the retrieved context actually correct in the world."
A pipeline retrieving from a stale index will score high on faithfulness while confidently generating wrong answers about current facts. The judge scores the output against the retrieved context. The retrieved context is what the retrieval layer returned. Nothing in the measurement stack checks whether the retrieval was working from up-to-date, authoritative source material.
User signals will eventually catch this because users know the current facts. But by the time enough user signal has accumulated to move the top-line accuracy number, the system has been producing wrong answers for days or weeks.
This is the trust-layer problem measurement alone cannot solve. Every mature production stack has to build custom infrastructure for source freshness verification, provenance tracking, and lineage-aware evaluation. The measurement layer is necessary. It is not sufficient.
Where to Start
If a team has never measured production accuracy, four steps get to a working baseline.
Add reference-free automated evaluation on a 5 percent random sample of production traffic. Faithfulness, answer relevance, task completion. Score every sampled trace. Aggregate into a running accuracy metric.
Instrument user feedback. Thumbs up and thumbs down at minimum. Retry rate at minimum. Escalation rate if the product supports it. Capture the signals into the same dashboard as the automated evaluation scores.
Add drift detection on input distribution and output distribution. Alert when either drifts more than 2 standard deviations from the baseline over any 24-hour window.
Set a monthly human-labeled sample of 200-500 traces to calibrate the automated evaluation. When the automated scores and the human labels start disagreeing systematically, the judge prompt needs revision or the automated evaluation is losing accuracy.
That is the minimum viable production measurement. Everything beyond it is refinement.
Structure before tools. Strategy before speed. Systems before scale.
Production is where the system meets the world. If you cannot measure how well they get along, you cannot know when they stopped.
The AI Operator Framework is the operator playbook for shipping AI work that compounds. Read the framework here or take the course.
