Relevance is whether an AI output or a retrieved result addresses the user’s actual request. A relevant answer stays on task and carries information that helps the person get what they came for. It is one of the oldest ideas in information retrieval and still one of the hardest to pin down, because it is a judgment about a need rather than a property of a document.
That distinction is the whole subject. A passage is not relevant on its own. It is relevant to someone, asking something, at a particular moment, with a particular amount of background knowledge. Change any one of those and the same text can go from useful to useless without a single word changing. This is why relevance resists a clean formula, and why teams that treat it as a solved measurement tend to be surprised by what ships.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Relevance is judged against the underlying need, not against the words of the query. Keyword overlap and vector proximity are proxies for it, not definitions of it.
- It is a separate axis from correctness, grounding, and safety. An answer can be fluent, factual, well-sourced, safe, and still miss the point.
- In an agent, relevance applies to more than the final text: retrieved documents, selected tools, and planned actions can each be on or off target.
- It is often graded rather than binary, because “partly useful” is a real and common state that a yes-or-no label destroys.
- Annotator disagreement on relevance is normal, not a process failure. A written rubric is what makes the judgment repeatable.
Relevance is about the need, not the query text
A query is a lossy encoding of an intent. Someone types six words standing in for a question they have not fully articulated, and the system has to recover the rest. When retrieval matches the six words instead of the intent, you get results that are topically correct and practically worthless.
The failure has a recognizable shape. A user asks how to cancel an annual plan and gets the page about upgrading an annual plan: same product, same vocabulary, wrong question. Every surface signal says match, and the need is unmet. Any measurement built purely on textual or embedding similarity scores that result well, which is why relevance has to be evaluated directly rather than inferred from a similarity score.
Underspecified queries make it harder still. “How do I fix this error” is relevant to whatever error the user is looking at, and the system cannot know that unless the surrounding conversation supplies it. Relevance in a multi-turn system is judged against the accumulated context, not the last message alone.
What relevance is not
Keeping these apart is most of the practical value of the term.
Not correctness. An answer can be squarely on topic and factually wrong. It can also be impeccably accurate about something the user did not ask.
Not grounding. A response can cite real retrieved sources faithfully and still answer a neighboring question. Grounding asks whether the claims trace back to the provided evidence. Relevance asks whether any of it was the point. Examples of how LLM outputs go wrong make the split concrete: some failures are invented facts, others are true statements aimed at the wrong target.
Not safety or tone. A refusal can be the correct and safe output while being entirely irrelevant to what was asked.
Not the same as its narrower measurements. In RAG evaluation, the per-chunk question of whether a retrieved passage helps answer the query has its own name, context relevance, and its own scoring conventions. General relevance is the broader property that measurement is one instance of.
Where relevance shows up in an agent
For a single-turn model, relevance is a question about one output. For an agent it applies at several points, and it can fail at any of them while looking fine at the others:
- Retrieved documents. Did the search return material that bears on the request?
- Tool selection. Was calling the billing API the right move for a password question?
- Plan steps. Does this step advance the user’s goal or a goal the agent invented?
- Final response. Does the answer address what was asked, in the form it was asked for?
An agent that retrieves perfectly, picks the wrong tool, then writes a confident summary of the wrong result produces an irrelevant outcome from mostly relevant parts. Locating which step went off target requires those intermediate steps to be recorded, one reason an agent observability platform captures tool calls and retrieval results rather than input and output alone.
Judging relevance consistently
Relevance is subjective in a specific and manageable way. Two careful annotators will disagree on borderline cases, and the fix is not to find better annotators but to narrow the question. A usable rubric says what the user was trying to accomplish, what counts as partially relevant, and how to handle results that are correct but answer an adjacent question.
Once the rubric exists it can be handed to a model instead of a person for scale, with the caveat that applies to any automated judgment: it needs validating against human labels first. The guidance on when a judge model is appropriate matters more here than for objective checks, because a subjective criterion gives the judge more room to drift.
Graded scales usually beat binary ones for relevance. A 0 to 3 scale keeps the distinction between “answers the question,” “useful but incomplete,” and “on topic but useless,” and that middle ground is where most real disagreement lives.
FAQ
What is the difference between relevance and accuracy?
Relevance asks whether the output addresses the request. Accuracy asks whether it is correct. They are independent: an accurate answer to the wrong question fails on relevance, and a well-targeted answer can still be factually wrong. Most evaluation suites score them separately for exactly that reason.
How do you measure relevance?
Through judgment applied to a fixed set of examples, either by humans working from a rubric or by a model instructed with the same rubric. Ranking metrics such as Precision@K, MRR, and NDCG then aggregate those per-result judgments into a score for a retrieval system. Relevance itself is the label; those metrics are ways of summarizing many labels.
Is relevance binary or graded?
Both are used. Binary labels are cheaper and simpler to aggregate, and some metrics require them. Graded labels preserve the partial cases that binary labels flatten, and they are required for NDCG. Pick based on whether partial usefulness changes what you would do.
Can a response be relevant but unhelpful?
Yes, and this is common. A response can address the right topic without being specific enough, complete enough, or actionable enough to be worth reading. Relevance is a necessary condition for a good answer, not a sufficient one, which is why it is usually scored alongside completeness and correctness.
Why do annotators disagree about relevance?
Because they are inferring an unstated need from a short query, and reasonable people infer differently. Disagreement concentrates on results that are topically adjacent. Measuring that disagreement rather than hiding it is useful: a criterion that humans cannot apply consistently is one a model will not apply consistently either.
