Mean reciprocal rank, or MRR, measures how high the first relevant result appears in a retrieved ranking. For each query, the reciprocal rank is 1 divided by the position of the first relevant result. If the first relevant result is ranked second, the reciprocal rank is 0.5. MRR is the mean of those reciprocal ranks across a set of queries.
The metric exists because in many systems only the first good hit matters. A user scanning a results page usually stops at the first item that answers the question. MRR puts a number on that one behavior and ignores nearly everything else, which is both why it is useful and why it is easy to misread.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- MRR is the mean of
1 / rankacross a query set, where rank is the position of the first relevant result. A query with no relevant result contributes 0. - The metric looks only at the first relevant result. Every result below it, relevant or not, has no effect on the score.
- Its discount is steep at the top. Moving a result from position 2 to position 1 adds 0.5 to that query’s score, while moving it from position 10 to 9 adds about 0.011.
- MRR assumes binary relevance. When your labels are graded, NDCG keeps information MRR discards.
- A single MRR number hides its own distribution, so report the share of queries scoring 0 next to the mean.
The formula
For a query set Q, MRR is the average of 1 / rank_i over every query i, where rank_i is the position of the first relevant result. Positions are 1-indexed, so a first-place hit scores 1.0. By convention, a query with no relevant result inside the cutoff k scores 0 rather than being dropped. That convention matters more than it looks: dropping misses instead of scoring them 0 raises your MRR every time retrieval fails completely, which is the opposite of what a retrieval metric should do.
A worked example
Four queries against the same retriever, top 10 results each, relevance already labeled.
| query | position of first relevant result | reciprocal rank |
|---|---|---|
| how do I rotate an API key | 1 | 1 / 1 = 1.0 |
| billing contact for the enterprise plan | 2 | 1 / 2 = 0.5 |
| why did my export time out | 4 | 1 / 4 = 0.25 |
| SSO with Okta group mapping | none in top 10 | 0 |
Sum the reciprocal ranks: 1.0 + 0.5 + 0.25 + 0 = 1.75. Divide by the four queries: 1.75 / 4 = 0.4375.
Resist one tempting misreading. Do not invert the score to recover an average position. The mean of reciprocals is not the reciprocal of the mean, so 0.4375 does not mean the first relevant result sits around position 2.29. It means one query was perfect, two were serviceable, one failed outright, and those outcomes average to 0.4375 on this scale.
Why the top of the list dominates
The reciprocal sequence drops fast and then flattens: 1.0 at position 1, 0.5 at 2, about 0.333 at 3, 0.25 at 4, 0.2 at 5, and 0.1 at 10. Almost all of the metric’s range lives in the first three slots.
That shape makes MRR a natural scoreboard for reranking, where a second-stage model reorders a candidate set that first-stage retrieval already returned. Approaches such as listwise document reranking with open-source models target exactly that job: not finding more relevant documents, but moving the ones you have into a better order. If a reranker earns its latency cost, MRR is where you see it first. The flip side is that the metric is nearly blind past position 5, so a change that reshuffles results 6 through 20 can genuinely help users and barely move the score.
What MRR does not tell you
This is the limitation to say out loud, because it is the one that burns teams. MRR only looks at the position of the first relevant result.
Take two retrievers scored on the same query. System A returns a relevant document at position 1 and nothing else relevant in the top 10. System B returns relevant documents at positions 1, 2, and 3. Both have a reciprocal rank of 1.0, so their MRR is identical. If you are filling a context window with five chunks, System B is substantially better and the metric cannot see it.
The same blindness applies to coverage. A query with one correct answer and a query with forty relevant documents score the same way, so MRR says nothing about how much relevant material you surfaced.
Two neighboring metrics fill those gaps. Precision@k asks what fraction of the top k results are relevant, capturing how much good material is up there but ignoring the order. NDCG uses graded relevance labels and applies a positional discount across the whole top k, rewarding both good documents high and several of them.
Computing MRR you can trust
MRR needs a relevance judgment per result, whether from humans, click data as a cheap and biased proxy, or a judge model. It also needs a stated cutoff, since MRR over the top 10 and MRR over the top 100 are different metrics.
The query set then has to stay frozen, because comparing this week’s MRR to last week’s is only valid if the queries did not change. That is why retrieval metrics usually live inside an evaluation harness instead of a notebook someone reruns by hand. Offline scores still drift as the corpus grows, so recomputing MRR on real traffic means recording retrieved document IDs and positions on the retrieval span, a standard part of tracing and evaluating a RAG or agent pipeline.
FAQ
What does MRR stand for in AI?
Mean reciprocal rank. Worth flagging that the same three letters mean monthly recurring revenue in a business context. In machine learning, information retrieval, and LLM evaluation, MRR always means mean reciprocal rank.
What is the mean reciprocal rank formula?
Take the position of the first relevant result for each query, invert it, and average those values across all queries. A query with nothing relevant inside the cutoff contributes 0 to the sum but still counts in the denominator.
What does MRR emphasize in retrieval evaluation?
The ranking position of the first relevant retrieved result, and nothing else. It does not measure how many relevant documents came back, how they are ordered among themselves, or how strongly relevant they are.
What is a good MRR score?
No threshold transfers across systems. The score depends on the cutoff, how hard the query set is, and how strictly relevance was labeled. An MRR of 0.6 on adversarial queries at k = 5 can reflect a better retriever than 0.8 on easy queries at k = 100. Treat it as a relative measure of whether a change helped.
How is MRR used in RAG systems?
It scores retrieval in isolation, before generation. If MRR is low, the model is being handed context that buries the answer, and no amount of prompt work fixes that. Separating retrieval quality from generation quality is the main reason to compute it.
