Context relevance measures whether the retrieved or supplied context is useful for answering the query. A context chunk can be semantically similar to the query but still not contain the information needed to answer correctly.
In RAG evaluation, context relevance helps distinguish retrieval noise from generation failure. If context relevance is low, improve retrieval, chunking, filters, query rewriting, or embeddings before tuning the final prompt.
It is a narrow measurement on purpose. One question, one piece of retrieved context, one judgment: does this text help answer that question. Nothing about whether the final answer was right, and nothing about whether anything was missing.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Context relevance is scored per chunk against the question, which makes it a precision-style measure of how much of the context is signal.
- Vector similarity and relevance are different things. A passage about the same topic that answers a different question scores high on similarity and zero on relevance.
- It is reference-free, so it runs on live traffic where no golden answer exists.
- A high score does not mean retrieval succeeded. Every returned chunk can be relevant while the one chunk that completes the answer is still missing.
- Low context relevance is a retrieval instruction, not a prompt instruction. Prompt tuning cannot repair a context window full of near-misses.
Similarity is not relevance
An embedding score measures how close two vectors are, which is a proxy for topical resemblance, and it degrades in predictable ways.
A question asking “how do I cancel an annual plan” will happily retrieve the passage describing how to upgrade an annual plan. Same product, same vocabulary, same tone, and it does not answer the question. A judge reading the pair says no. The cosine score says 0.89.
The gap widens for negation, temporal qualifiers, and specific entities. “Which regions are not covered” and “which regions are covered” embed almost identically, and so do “2024 policy” and “2025 policy” when the documents are otherwise the same. Context relevance catches what the similarity score cannot see.
How it gets scored
The standard instrument is a judge model given the question and one retrieved chunk, asked whether the chunk contains information that helps answer the question. No reference answer is involved, which is why this works on production traffic.
Three choices shape whether the number means anything:
Scale. Binary relevant or not relevant is more reproducible than a 1 to 5 rating, where nobody, including the judge, can defend the difference between a 3 and a 4 across runs. If you need more granularity, three levels tend to hold: directly answers, related background, not relevant.
Unit. Scoring each chunk separately gives a per-request precision figure and shows where in the ranking the noise starts. Scoring the assembled context as one block is cheaper and tells you less.
Rubric. “Relevant” is underspecified. Decide explicitly whether background that supports the answer without containing it counts, and write that into the prompt with examples. Most judge and human disagreement on this metric traces back to an unstated definition rather than model capability.
Calibrate before trusting the trend. Hand-label a couple hundred question and chunk pairs and check agreement on the negative class in particular, since judges tend toward generosity. Deciding when an LLM judge is the right tool comes before tuning its prompt. For borderline calls, where reasonable humans disagree, aggregating several models in a jury arrangement is steadier than pushing one judge to be decisive.
What the score tells you, and what it does not
Low context relevance means the model is working with mostly noise. Irrelevant passages consume the token budget a needed passage could have occupied, and a model given nothing that answers the question will often answer anyway, from a near-miss passage or from memory, producing the confidently sourced wrong answer that RAG hallucination benchmarks are built to measure.
High context relevance guarantees less than it appears to. Every chunk can be relevant while the set is incomplete: three passages that each partially address a comparison question, none containing the number you needed. That blind spot is why context relevance is one input to retrieval quality rather than a substitute for it.
It also says nothing about what happened next. The model can be handed perfectly relevant context and still misread it. That is faithfulness, measured against the same context but on the generation side. Pairing the two is what makes the diagnosis unambiguous: relevant context plus an unfaithful answer is a generation problem, irrelevant context plus a wrong answer is a retrieval problem.
Where it sits among neighboring measures
Retrieval quality is the property of the retrieval step overall, covering both the noise in what came back and whether anything needed is missing. Context relevance is the first of those two, measured per chunk. Answer relevance is the mirror image on the output side: did the response address the question asked. Faithfulness compares the answer to the context, while context relevance compares the context to the question. Different comparisons, different failures, and the reason RAG evaluation scores them separately.
FAQ
What is the difference between context relevance and answer relevance?
Context relevance looks at the input to generation, answer relevance at the output. Context relevance asks whether the retrieved passages help answer the question. Answer relevance asks whether the response addressed what was asked instead of drifting to a neighboring topic. They fail independently: a well-targeted answer can be produced from weak context by a model filling gaps from memory, and rich, relevant context can still yield an answer to the wrong question.
Is contextual relevance the same thing?
Yes. Contextual relevance, contextual relevancy, and context relevance all name the same measurement, and eval frameworks differ on which label they print. What varies between implementations is the unit of scoring, whether each chunk is judged individually or the assembled context is judged as a whole, and that difference does change which numbers are comparable.
How do I measure context relevance without labeled data?
That is the normal case, and the metric is built for it. A judge model sees the question and the retrieved chunk and decides whether the chunk helps. No ground-truth answer and no corpus-wide relevance labels are required, which is what lets it run on sampled production traffic. The tradeoff is that you are trusting a model’s judgment, so spend the effort on a small human-labeled calibration set.
Should every retrieved chunk be relevant?
No, and forcing it is usually a mistake. Pushing per-request context relevance toward 1.0 means retrieving fewer chunks, which raises the chance of dropping the passage that completes an answer. Some dilution is the price of coverage. The useful signal is a drop over time, or a cluster of requests where relevance collapses to near zero, which almost always points at a content gap for that question type.
Context relevance is low. What do I change?
Work upstream in order. Check that the answering document exists in the index and is current. Look at whether the query sent to the retriever was a standalone question or a conversational fragment. Add a reranker if the right passages come back ordered badly. Then look at chunk boundaries, since chunks that split an idea in half produce fragments judges correctly score as unhelpful. Prompt changes come last, and only to control how the model behaves when the context genuinely does not answer the question.
