Precision@K measures how many of the top k retrieved results are relevant. If a retriever returns five documents and three are relevant, Precision@5 is 0.6. It is written Precision@k, where k is the cutoff you chose, and it is always reported for a specific k because the value changes as the cutoff moves.
The metric earns its place in RAG evaluation because context window space is finite and irrelevant context is not free. Every document that makes the cut displaces one that could have been there, and irrelevant documents can confuse the model that has to read them. High recall with low precision still hurts: the right answer is in the prompt, buried among four things that look almost as relevant.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
Precision@kis the count of relevant results in the topkdivided byk. Nothing below the cutoff affects it.- It ignores the order of results inside the cutoff, which is its defining limitation and occasionally its point.
- The maximum score is capped when fewer than
krelevant documents exist for a query, so a low score is not always the retriever’s fault. - Precision and recall trade against each other as
kmoves, so report both, or report precision at thekyour pipeline actually uses. - It needs only binary relevance labels, which are cheaper to produce than the graded labels NDCG requires.
The formula
Count the relevant results in the top k, then divide by k. That denominator is always k, not the number of results returned and not the number of relevant documents in the corpus. If your retriever returns three documents but you measure at k = 5, you still divide by 5.
Relevance here is binary. A document counts or it does not, with no partial credit for something topically adjacent that fails to answer the question. When that all-or-nothing judgment feels wrong for your data, that is the signal to look at NDCG instead.
A worked example
One query, five results, each labeled relevant or not.
| position | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
| label | not relevant | relevant | not relevant | relevant | relevant |
Precision@1: 0 relevant in the top 1, so 0 / 1 = 0.0Precision@3: 1 relevant in the top 3, so 1 / 3 = 0.333Precision@5: 3 relevant in the top 5, so 3 / 5 = 0.6
Three numbers from one ranking, disagreeing about how good it is. That is not a defect. Precision@1 says the top slot is wasted, Precision@5 says most of the list is usable, and both are true. Which one matters depends on how many documents your pipeline consumes.
Order inside the cutoff does not matter
This is the part that surprises people. Take two rankings of the same five documents:
- List A: relevant, relevant, relevant, not relevant, not relevant
- List B: not relevant, not relevant, relevant, relevant, relevant
Both hold three relevant documents in the top 5, so both score Precision@5 = 3 / 5 = 0.6. Identical. Yet List A is clearly better for a human scanning results, and its reciprocal rank is 1.0 against List B’s 0.333.
Whether that blindness is a problem depends on the consumer. If a model reads all five chunks into its context, composition matters more than order and precision measures the right thing. If a person scans a results page, or the pipeline truncates to whatever fits, order matters a lot and precision alone will mislead you.
Two adjacent metrics cover the gap. MRR looks only at the position of the first relevant result, so it captures ordering at the very top but says nothing about how many relevant documents you found. NDCG uses graded labels and applies a positional discount across the whole top k, accounting for both composition and order at the cost of more expensive labels.
Precision, recall, and the ceiling problem
Recall@k asks what fraction of all relevant documents in the corpus reached the top k. Using the example above, if eight relevant documents exist and three were retrieved, Recall@5 is 3 / 8 = 0.375 while Precision@5 is 0.6.
The two move in opposite directions as you widen k. Raising k can only hold recall steady or improve it, and it usually drags precision down, because documents ranked sixth through twentieth are ranked there for a reason.
There is also a ceiling that catches teams out. If only two relevant documents exist for a query, the best possible Precision@5 is 2 / 5 = 0.4. A perfect retriever scores 0.4 on that query. Averaging it with queries that have twenty relevant documents produces a number that describes your query set more than your retriever.
Making the number trustworthy
Choose k to match reality. If your prompt template inserts four chunks, measure at k = 4. Reporting Precision@10 for a pipeline that uses three documents measures a system you did not build.
Labels are the harder half. Binary relevance sounds simple until two annotators disagree about a passage that is on topic but contains no answer. Write the rubric down first, and if a model is generating labels, treat that judge as something to validate rather than trust, since the guidance on building and calibrating an LLM judge applies to relevance labeling as much as to scoring final answers.
Then freeze the query set, because a retrieval metric that moves when the queries change tells you nothing. That is why these scores belong in a repeatable suite alongside the other checks in AI agent testing. The same computation runs against production traffic once retrieved document IDs and positions are recorded on the retrieval span, standard practice when tracing and evaluating a RAG pipeline. For guidance on testing agent behavior, see agent evaluation.
FAQ
What is the Precision@K formula?
Divide the number of relevant results in the top k by k. The denominator is the cutoff, not the number of documents returned, so a short result list is penalized rather than ignored.
What is the difference between Precision@K and Recall@K?
Precision asks what fraction of what you returned is relevant. Recall asks what fraction of everything relevant you managed to return. Precision is about the cost of noise in the context window, recall is about the cost of a missing answer. Most retrieval work is a negotiation between the two.
What is a good Precision@K score?
No portable threshold exists. The score depends on k, on how many relevant documents each query has, and on how strictly relevance was labeled. Use it to compare two versions of the same retriever on the same query set.
How do I choose k?
Match it to how many retrieved documents your application actually consumes. For RAG that is usually the number of chunks the prompt template inserts. Reporting several values, such as Precision@1, Precision@3, and Precision@10, also works when you want to see how quality decays as the list grows.
Does Precision@K work with graded relevance?
Not directly. It needs a binary decision, so graded labels have to be collapsed with a threshold, which throws away the difference between a perfect match and a partial one. If that distinction matters, NDCG is the metric built for it.
