How to Evaluate a RAG System

Most teams evaluate RAG the wrong way.

They score the answer. The answer looks right. The score is high. The system ships. Three weeks later, users start reporting failures the eval never caught.

The reason is almost always the same. The evaluation measured the generator and ignored the retriever. The generator was answering coherently from the wrong context. The score stayed high. The user got the wrong answer.

A RAG system has two stages. A real evaluation covers both.

The Two Stages a RAG System Has to Get Right

Retrieval-augmented generation is two systems stitched together. The retriever pulls documents. The generator writes the answer using those documents. Each stage can fail independently, and each failure looks different in production.

The retriever fails when it pulls the wrong chunks, misses a chunk that contains the answer, or ranks the good chunks below noise. The generator sees whatever the retriever handed it and writes coherent prose from that context. If the context was wrong, the answer is wrong, but the answer still reads like a confident response.

The generator fails when it fabricates content the retrieved context does not support, ignores the retrieved context and answers from parametric memory, or answers the wrong question. Any of these produces an output that fails downstream even if the retrieval was correct.

You cannot debug a RAG failure without knowing which stage produced it. That is the whole point of evaluating them separately.

What to Measure at the Retrieval Stage

The retrieval stage produces a ranked list of chunks. The evaluation asks two questions of that list. Did the right chunks come back at all. Were they ranked correctly.

Context recall. Of the chunks needed to answer the question, how many did the retriever surface. A recall of 1.0 means every necessary chunk was retrieved. A recall of 0.5 means half the needed context is missing. Low recall is the failure mode where the generator has to guess or fabricate because the evidence is not in the context window.

Context precision. Of the chunks the retriever returned, how many are actually relevant. High precision means the retrieved set is clean. Low precision means the retrieved set is noisy, which forces the generator to sort useful signal from irrelevant filler. A retriever can have perfect recall and terrible precision by returning every chunk in the corpus.

Precision at k, recall at k, MRR, NDCG. Classical information retrieval metrics that measure ranking quality. Precision at k measures how many of the top k results are relevant. Mean Reciprocal Rank measures how far down the list the first relevant result appears. Normalized Discounted Cumulative Gain weights the relevance of each result by its position. When the retriever surfaces the right chunks but buries them at position eight, MRR and NDCG catch that. Simpler metrics do not.

Every mature RAG evaluation setup tracks at least one metric per stage. Tracking only recall misses ranking failures. Tracking only precision misses coverage failures. A complete retrieval-stage picture combines context recall, context precision, and one of the ranking metrics.

What to Measure at the Generation Stage

The generation stage produces the final answer. The evaluation asks three questions of that answer. Is it supported by the retrieved context. Does it address the actual question. Is it complete.

Faithfulness. The most-used generation-stage metric. Every claim in the generated answer is checked against the retrieved context. The score is the ratio of claims supported by context to total claims. A faithfulness of 1.0 means every claim in the answer traces to the retrieved chunks. Anything below 1.0 means the generator invented content the retrieval did not support. That invented content is what most people call hallucination.

Answer relevance. Does the answer actually address the question that was asked. A high-faithfulness answer can still be a bad answer if it responds to a different question than the one the user typed. Answer relevance is measured against the query, not against the context.

Answer correctness. Does the answer match a known-correct answer for this question. This requires a ground-truth answer to compare against. When you have that ground truth, correctness is the strongest single signal. When you do not, correctness is estimated by proxy through faithfulness and relevance combined.

Faithfulness plus answer relevance is the most-common generation-stage pairing. Faithfulness catches fabrication. Answer relevance catches off-topic responses. Together they cover the two most common ways a generator produces a bad answer even when the retriever did its job.

The Gold-Standard Test Set Is the Foundation

Everything above assumes you have a test set. Question, retrieved context, correct answer. Without that, none of these metrics can be computed.

Building the test set is the highest-impact evaluation work. It is also the work most teams skip because it is unglamorous. The right approach depends on where you are in the RAG system lifecycle.

Early stage. You do not have production traffic yet. Build the test set by hand from real questions the target users would ask. Sit with a subject-matter expert. Have them ask thirty questions. Have them write the correct answer for each. Have them identify which chunks in your corpus contain the answer. This is the ground-truth dataset. Thirty examples is enough to start. A hundred examples is meaningfully better.

Growth stage. You have production traffic but not yet enough failures to learn from. Use synthetic generation to expand the seed dataset. Ragas, DeepEval, and most modern evaluation frameworks can generate synthetic question-answer pairs from your corpus using an LLM. The output requires human review because LLMs generate plausible-looking test cases that contain factual errors. Never trust synthetic data unreviewed.

Production stage. The best test set comes from real user interactions. Every failing production query is a potential test case. Every user correction is a signal about where the system is weak. The mature workflow captures failing production traces, converts them into evaluation cases, adds them to the test set, and runs regression testing on every model or prompt change. The test set grows with the system.

None of these test-set approaches are exclusive. A production-grade RAG evaluation combines all three. A curated seed set for baseline coverage. Synthetic augmentation for edge cases. A growing production-derived set for real-world regression.

The Frameworks That Do the Work

You do not need to build any of this from scratch. Several frameworks compute the standard metrics against a test set.

Ragas. The most-adopted open-source framework. Started as a 2023 research paper. Ships the four core metrics (context precision, context recall, faithfulness, answer relevance) plus more than twenty others. Integrates with LangChain, LlamaIndex, Haystack. Programmatic. Add it to your codebase, feed it your evaluation traces, get back scored results. Ragas focuses on the metrics, not the surrounding workflow. Teams pair it with tracing and observability tools.

DeepEval. Brings the unit testing model to LLM evaluation. Every eval is a test case with pass or fail criteria. Integrates with pytest so evaluation runs in CI alongside traditional software tests. If your team already treats evals as code that must pass before merge, DeepEval fits that shape.

TruLens. Ships the RAG Triad. Three metrics for three stages: context relevance (question to retrieved context), groundedness (answer to retrieved context), answer relevance (answer to question). Compact and legible. Best for teams that want a clear conceptual model of what to measure.

Arize Phoenix. Open-source observability that also does evaluation. Best for teams that want tracing, spans, and evals in one tool without paying for a hosted platform.

Braintrust, LangSmith, Confident AI, Maxim. Commercial platforms that combine evaluation, experimentation, and production monitoring. Best for teams that want a single dashboard, want to convert production traces into eval datasets with one click, and want the observability layer to feed the evaluation layer automatically.

The choice between them is mostly about workflow, not about metrics. All of them compute similar underlying metrics. What differs is how the metrics get into your CI pipeline, how failing production traces become test cases, and how much of the workflow is your problem versus the platform's problem.

Offline Evaluation vs Production Monitoring

Two different jobs. Frequently confused.

Offline evaluation. Runs against a fixed test set. Deterministic. Repeatable. Used during development, in CI, and for regression testing after prompt or model changes. The output is a snapshot of how the system performs on the test set. If the system passes, the change is safe to ship.

Production monitoring. Runs against live traffic. Not deterministic. Not exhaustive. Used to catch drift after deployment. Sample a percentage of production requests. Compute the eval metrics on the samples. Track the trend line over time. Alert when metrics fall below the baseline.

Offline evaluation tells you whether a change is safe. Production monitoring tells you whether the world changed. Both are necessary. Neither substitutes for the other. Most teams have one and miss the other, and the one they miss is most likely production monitoring.

The Failure Mode That Every Team Underestimates

Context trustworthiness. The dimension that standard metrics do not cover.

The four core metrics assume the retrieval index is trustworthy. Faithfulness measures whether the answer reflects the retrieved chunks. It does not measure whether the retrieved chunks reflect current, authoritative source material. A RAG system scoring 0.95 faithfulness against a stale index will confidently generate answers based on outdated content. The eval will not catch it because the eval measured the wrong thing.

This is the failure mode where legal RAG systems cite repealed statutes with high faithfulness. Where medical RAG systems recommend deprecated protocols with high answer relevance. Where product-support RAG systems tell customers about features that were removed six months ago.

Trustworthiness is a source-side problem, not a generation-side problem. The evaluation loop that catches it is different. Every retrieved chunk should carry metadata about its source, its last-verified date, and its ownership. When that metadata drifts out of date, the retrieval index needs refresh regardless of what the standard metrics say.

This is the failure I keep coming back to. It is what most current evaluation frameworks under-address. It is what production teams eventually hit and have to build custom infrastructure to solve.

Where to Start

If you are evaluating a RAG system for the first time, three steps get you to a real baseline.

One. Build a seed test set by hand. Thirty questions with correct answers and correct source chunks. Do not use synthetic data yet. The manual work teaches you what the system is actually being asked to do.

Two. Wire Ragas or DeepEval into your test suite. Compute context recall, context precision, faithfulness, and answer relevance on every eval run. Baseline the current scores. Do not chase specific numbers yet. You are establishing a floor.

Three. Add a production sampling layer. Pick 5% of production traffic. Log every retrieval and generation step. Score the samples on the same four metrics. Track the trend line. Alert on regression.

That is the minimum viable RAG evaluation. Everything beyond it is refinement. Human review of a percentage of samples. Synthetic augmentation of the test set. Category-specific metrics for domain-critical concerns. Custom evaluators for regulated content. Trust-layer measurement for source freshness.

The teams that ship reliable RAG systems in production have this loop running. The teams that ship RAG systems that fail have skipped it.

Structure before tools. Strategy before speed. Systems before scale.

Evaluation is the layer between building AI and trusting AI. Most operators skip it. The ones who do not skip it are the ones whose AI systems stay in production.

The AI Operator Framework is the operator playbook for shipping AI work that compounds. Read the framework here or take the course.