Metric selection is the highest-impact evaluation decision most teams make without much thought.

The pattern is familiar. A team decides to add LLM evaluation. They open Ragas or DeepEval and find a menu of 20 to 50 available metrics. They pick five that sound important. They wire them up. They watch the dashboards. Six months later they realize the metrics they picked do not correlate with the failures their users complain about. The system was scoring 0.9 on their chosen metrics while shipping the exact answers users were escalating.

The metric set is a strategic decision, not a tooling decision. Different applications need different metrics. Getting the set right is what makes evaluation predict production quality instead of predict its own scores.

The Metrics That Matter Most in Production

Six metrics show up in almost every mature production evaluation stack. Every application weights them differently, but the set is remarkably consistent across teams that ship reliable systems.

Faithfulness. The share of claims in the output supported by the retrieved context or the source material. This is the metric that catches fabrication. Any system that generates from retrieved evidence needs this at the top of the list.

Groundedness. A close cousin of faithfulness, applied to the connection between the output and the input material. Where faithfulness asks "is this claim supported," groundedness asks "does the output stay within what the input authorized." Applications that transform inputs into structured outputs measure groundedness more than faithfulness.

Answer relevance. Whether the output actually addresses the question that was asked. A groundedness-perfect answer to the wrong question is still a bad answer. This metric is measured against the query, not against the context.

Task completion. Whether the system accomplished what the user was trying to do. For a support agent: was the ticket resolved. For a coding agent: did the code compile and pass tests. For a summarization system: did the summary cover what needed to be covered. This is the closest single metric to user satisfaction.

Latency at p95. How long the slowest 5 percent of requests take. Users experience the p95, not the average. A system with a 2-second average and a 30-second p95 feels slow, no matter what the mean says.

Cost per outcome. The dollar cost of producing one successful output. Not cost per request, cost per successful outcome. A cheap system that produces wrong answers requires retries and human interventions that dominate the true cost. Cost per outcome is what actually matters for scale economics.

These six form the metric spine for most production LLM systems. Applications add specialized metrics on top for domain-specific concerns.

The Metrics That Are Theater

Three categories of metrics look important in dashboards and do not predict production quality.

Metrics that measure the model instead of the system. MMLU. GPQA. HumanEval. These are benchmarks for model selection, not signals about whether your specific system works. Teams that put these on their production dashboard confuse "which model is smart" with "does our application work." Section headline on this failure mode: the leaderboard is not your app.

Metrics that are highly correlated with metrics you already track. If faithfulness and groundedness always move together in your application, tracking both is redundant. Pick one, drop the other, save the display space. Redundant metrics dilute attention on the metrics that carry independent signal.

Metrics that measure surface features instead of quality. Output length. Presence of specific keywords. Grammatical correctness. These are easy to compute. They rarely predict whether users found the answer useful. When a team's metric dashboard is heavy on surface features and light on outcome quality, the dashboard is optimizing for what is easy to measure, not what matters.

How the Application Shape Changes the Metric Set

Different LLM applications need different metrics. Same six-metric spine, different weighting, different specialized additions.

RAG applications. Faithfulness and context recall lead. Groundedness is baseline. Answer relevance keeps the system on-topic. Add context precision to catch retrieval ranking failures. Add source freshness as a custom metric to catch stale-index issues that standard evaluators miss.

Agent applications. Task completion leads. Tool-call accuracy is essential. Add a plan-quality metric that scores whether the agent's reasoning path made sense. Add a step-count metric to catch runaway agents that loop on themselves. Latency at p95 matters more than for RAG because agents accumulate latency across many steps.

Summarization systems. Faithfulness leads (catch fabricated content). Completeness leads equally (catch missing key points). Redundancy is a useful negative metric that catches summaries that repeat the same points multiple times. Length calibration matters if the system has to produce summaries in specific size targets.

Structured extraction systems. Groundedness leads (was the extracted value in the source). Add field-level accuracy that scores each extracted field against ground truth separately. Add schema compliance that catches outputs that violate the expected structure. Cost per document dominates the operational metric set.

Classification systems. Traditional ML metrics still work. Accuracy, precision, recall, F1. Confusion matrix reveals which classes the system confuses. Calibration score reveals whether the confidence outputs are trustworthy. Add human-in-the-loop metrics that measure when the system correctly escalates versus incorrectly self-serves.

Conversational systems. Answer relevance leads. Add conversation coherence that catches contextual drift across multiple turns. Add resolution rate that measures whether conversations end in successful outcomes. User satisfaction scores from post-conversation surveys, when available, are the ground truth to correlate other metrics against.

The Metric No One Talks About Enough

Cost per outcome is the metric that separates production-ready systems from prototypes.

Most teams track cost per API request. That number is easy to get from any LLM provider's dashboard. It is also misleading. A system that costs one cent per request and produces the wrong answer 30 percent of the time has a much higher true cost per successful outcome than a system that costs three cents per request and produces the right answer 95 percent of the time.

Cost per outcome requires two things most teams do not measure. First, the retry cost when a request fails and needs to run again. Second, the human intervention cost when a request produces an output that requires review. Both dominate true unit economics at scale.

The teams that get this right build cost per outcome dashboards that account for the full path from user request to successful outcome. Those teams make different model selection decisions, different prompt engineering decisions, and different infrastructure decisions than teams that only track cost per request.

The Metrics That Should Trigger Alerts vs Dashboards

Not every metric belongs on an alert. Alerts should fire on regressions that matter. Dashboards should show the full picture.

Alert-worthy metrics. Faithfulness drops below the shipping threshold. Task completion drops below the last two weeks' baseline. Cost per outcome exceeds the budget threshold. Latency at p95 exceeds the SLA. These are the metrics where a real change means user experience is degrading and someone needs to act.

Dashboard-worthy metrics. Trend lines for all six spine metrics. Distributions of scores across different user segments. Correlations between metrics that help debug which layer is failing. Cost breakdowns by model, prompt version, and application feature. These are the metrics that get looked at during weekly reviews and root-cause investigations.

Teams that alert on everything drown in noise. Teams that alert on nothing miss regressions until users complain. The right set is small: three to five alerts on the metrics that most directly predict user experience.

The Failure Mode Metrics Cannot Catch

Every metric in this post measures a property of the output relative to the input, the context, or a ground truth. None of them measure whether the underlying knowledge the system draws from is accurate in the world.

A retrieval system with a faithfulness of 1.0 against a stale index produces confident wrong answers about current facts. Every metric says the system is working. The user experience says the system is broken.

A knowledge base that has drifted out of date passes every internal consistency check. The evaluators score everything favorably. The world moved. The system did not know it had.

Trust of source material is the dimension standard evaluation metrics do not cover. Every mature production evaluation practice has to address it, most times with custom infrastructure the evaluation frameworks do not ship. The gap is real. It is where a lot of production AI systems fail even when their metrics say they should not.

Where to Start

If a team is picking metrics for the first time, four steps get to a working set.

Start with the six-metric spine. Faithfulness, groundedness, answer relevance, task completion, latency at p95, cost per outcome. These cover the ground floor for almost every application shape.

Add two to four application-specific metrics that match the shape of the system. Context recall for RAG. Tool-call accuracy for agents. Completeness for summarization. Field-level accuracy for extraction.

Pick three to five metrics from the combined set to alert on. Everything else lives in dashboards. Alerts should mean "someone needs to act." If a metric moves and nobody would act, it does not belong on an alert.

Review the metric set every quarter. Applications evolve. What mattered six months ago may not matter now. What did not exist six months ago may be the most important signal today. A static metric set becomes a stale metric set.

The teams that ship reliable LLM systems in production have this discipline. The teams that ship systems that fail have a metric set they picked once and never revisited.

Structure before tools. Strategy before speed. Systems before scale.

The wrong metric measured well is worse than no metric at all. It creates confidence that is not justified by reality.

The AI Operator Framework is the operator playbook for shipping AI work that compounds. Read the framework here or take the course.