RAG evaluation measures how well a retrieval-augmented generation system retrieves relevant context and generates grounded answers from it. It usually includes retrieval metrics, context relevance, answer correctness, faithfulness, and citation or grounding checks.
Good RAG evals separate the pipeline into stages. Did the retriever find the right documents? Did the model use them? Did the answer stay faithful to the evidence? A single final-answer score usually does not tell you which part broke.
That last point is the whole discipline. Everything else on this page is a consequence of it.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Score stages, not just outputs. A thumbs-down on the final answer is a bug report with no stack trace.
- Three questions cover most of it: was the evidence retrieved, was it used, and did the answer stay inside it.
- Retrieval-side and generation-side failures look identical to the user and need opposite fixes.
- Most useful RAG evals are reference-free, so they run on production traffic where no golden answer exists.
- Agentic RAG makes several retrieval calls per question, moving the unit of evaluation from one query to a whole trace.
Why one score is not enough
Take a support assistant that answers a billing question wrong. Marking the answer incorrect tells you nothing actionable, because at least four distinct failures produce it:
- The refund policy was never indexed. Retrieval could not have worked.
- It was indexed but ranked eleventh, and the pipeline passes five chunks.
- It was retrieved, and the model answered from a memorized generic policy instead.
- It was retrieved and used, but the model overstated a conditional clause as absolute.
The first is an ingestion fix, the second a reranker or a larger k, the third and fourth prompt work. An aggregate accuracy number routes you to none of them, which is how teams end up spending a quarter tuning prompts against a retrieval problem.
The three stages worth scoring
Retrieval. Did the candidate set contain what was needed? With labeled data this is ranking measurement over known-relevant documents. Without labels it is a judge scoring each retrieved chunk against the question, which is what context relevance measures. The broader property belongs to retrieval quality, and the ways it breaks to retrieval failure.
Generation, given the context. This is faithfulness, sometimes called groundedness or context adherence: every claim in the answer is supported by the retrieved text and nothing was invented to fill a gap. It scores against the context rather than the truth, so it holds even when your corpus is wrong. Evaluating the generation stage separately is what isolates a model that ignores evidence from a retriever that never found it.
The answer itself. Correctness against a reference where you have one, plus answer relevance: did the response address the question asked rather than a neighboring one. Completeness belongs here too, since a technically true answer that omits the one condition the user needed is still a failure.
Faithfulness and correctness diverge more often than people expect. An answer can be perfectly faithful to a stale document and pass the faithfulness check while being wrong. That gap is a corpus problem, and only a reference-based check finds it.
Building the eval set
Start with production traces. Real questions expose the phrasing, ambiguity, and follow-ups a synthetic set never produces. Pull a few hundred, cluster by intent, and keep the hard clusters.
Synthetic questions generated from your own documents fill known coverage gaps, but they carry a bias: a question written from a chunk is answerable by that chunk, which makes retrieval look easier than it is. Mix them with real queries and label the source so you can read the two populations apart. Include questions your system should refuse, since a test set with no unanswerable questions in it silently rewards guessing.
Offline and online evaluation
Offline evaluation runs a fixed dataset through the pipeline whenever you change something: a new embedding model, a different chunk size, a reranker, a prompt edit. The point is comparison. Run both configurations against the same set and read per-stage deltas, because a change that lifts retrieval and drops faithfulness nets out to zero in an aggregate score while being clearly wrong.
Online evaluation runs judges against sampled production traffic and catches what offline sets cannot: corpus drift, seasonal question shifts, and the long tail of phrasings nobody wrote a test for. It depends on tracing being in place first, since the judge needs the retrieved chunks and the prompt as they actually were, which is the instrumentation covered in this guide to tracing and evaluating agents.
Calibrating the judge
An LLM judge is a model with its own error rate, and an uncalibrated judge is a confident source of wrong numbers. Label a couple hundred examples by hand and check agreement on the negative class in particular, since judges are generous by default. Binary labels with a written explanation hold up better than a 1 to 5 score, where the gap between 3 and 4 is not reproducible across runs of the same model. When agreement is poor, a sharper rubric usually beats a larger judge, and benchmarking work on RAG evals shows how much template choices alone move judge behavior.
FAQ
How do you evaluate a RAG agent?
An agent differs from a fixed pipeline because it decides when and what to retrieve, often several times and often with a rewritten query. Evaluate at two levels. Per retrieval call, score what you would in a pipeline: did the query make sense, and was the returned context relevant. Across the trace, score whether the agent retrieved at all when it should have, whether it stopped once it had enough, and whether the final answer used what it gathered. The agent-specific failures are retrieving once, getting partial information, and answering anyway, or looping on near-identical queries after a disappointing first result. Neither shows up in a per-call score.
What metrics should I use for a RAG system?
One retrieval metric, one faithfulness check, one answer-level check. With relevance labels, use a ranking metric for retrieval; without them, a context relevance judge. Add faithfulness against the retrieved context, and answer correctness where a reference exists, otherwise answer relevance. A tenth metric is worth less than making sure those three are calibrated and split by stage.
Do I need ground-truth answers to evaluate RAG?
No, and most production evaluation runs without them. Context relevance, faithfulness, and answer relevance are all reference-free: they compare the question, the retrieved context, and the answer to each other. Reference answers are still worth building for a small regression set, because they are the only way to catch a confidently faithful answer drawn from an outdated document.
How often should RAG evals run?
Offline sets run on every meaningful change to retrieval, prompts, or the model, the same way you would run tests. Online sampling runs continuously at whatever rate the budget allows, heavier right after a deploy or a corpus refresh. Watch for a growing cluster of low-scoring traces around one intent rather than small movements in the average, since that pattern is usually a content gap.
Why does my RAG system score well in testing and fail in production?
Usually the test set. Synthetic questions written from your own chunks are answerable by construction, and the eval never contained the ambiguous, multi-part, or unanswerable questions that dominate real traffic. The other cause is corpus drift: the index was fresh the day you built the set and has since fallen behind the documents it mirrors.
