Reranking is a second scoring step that reorders a list of retrieved results so the most relevant ones appear first. In retrieval-augmented generation (RAG) and agent systems, reranking helps the model see the right context, because only the top few results usually reach the prompt.
For example, suppose a help-center search retrieves 50 chunks for the query “cancel a prepaid annual plan.” The first ranking puts a monthly-plan cancellation guide at the top, and the prepaid annual guide sits at rank 7. A reranker scores all 50 chunks against the query and moves the prepaid annual guide to rank 1. The answer can now cite the refund rule it needs.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why reranking matters for RAG and agents
A first-stage retriever has to search a large index quickly. It is built for speed and recall, so it often returns the right chunk somewhere in the list but not at the top.
The generator usually sees only the top few chunks. If the right chunk is at rank 7 and the prompt includes the top 5, the model never sees it.
Reranking is useful for:
- Fixing buried results: Promote a relevant chunk that the retriever found but ranked too low.
- Handling subtle queries: Distinguish passages that share keywords but differ in meaning, such as monthly versus annual plans.
- Ordering merged results: Make a final, query-aware ordering after several retrievers are combined.
- Shrinking the context: Pass fewer, better chunks to the model.
For more on how retrieval fits into an answer pipeline, see retrieval-augmented generation (RAG).
How reranking works
Reranking splits retrieval into two stages. Each stage uses a different kind of model.
Stage 1: fast retrieval with a bi-encoder
A bi-encoder embeds the query and each passage separately. Passage vectors can be computed ahead of time, so searching a large index is fast.
The tradeoff is that the query and passage never interact during scoring. The model compares two summaries instead of reading them together. For more on this first stage, see embeddings and vector search.
Stage 2: careful scoring with a cross-encoder
A cross-encoder reranker takes the query and one passage as a single joint input. It outputs a relevance score for that pair. The Sentence Transformers retrieve and rerank example describes this pattern.
Reading the pair together handles negation and relationships between terms better. The cost is that the model must run once for every candidate at query time. That is why the candidate set is kept small.
Where reranking sits in the pipeline
In a RAG pipeline, the usual order is:
- Retrieve candidates from one or more retrievers.
- Optionally fuse the results from several retrievers into one list.
- Rerank the candidates.
- Pass the top results to the model.
Reranking after fusion reorders the merged list. Reranking before fusion changes what each list contributes. Record which stage you used in your logs, or debugging will confuse the two.
How to measure the effect of reranking
To measure reranking, compare the same candidate set before and after the reranker runs. Then check the final answer.
The following code finds the correct chunk’s rank in each list from the opening example. It reports the reciprocal rank and whether the chunk falls inside a top-5 context window. Only seven of the 50 candidates are shown.
def rank_of(doc_id, ranking):
return ranking.index(doc_id) + 1 if doc_id in ranking else None
def report(label, ranking, correct, k=5):
rank = rank_of(correct, ranking)
rr = 1 / rank if rank else 0.0
in_context = rank is not None and rank <= k
print(f"{label}: rank {rank}, reciprocal rank {rr:.3f}, in top {k}: {in_context}")
correct = "prepaid-annual-guide"
before = ["monthly-cancel-guide", "billing-faq", "plan-comparison", "refund-policy",
"account-settings", "upgrade-guide", "prepaid-annual-guide"]
after = ["prepaid-annual-guide", "refund-policy", "monthly-cancel-guide",
"billing-faq", "plan-comparison", "account-settings", "upgrade-guide"]
report("before rerank", before, correct)
report("after rerank ", after, correct)
Output:
before rerank: rank 7, reciprocal rank 0.143, in top 5: False
after rerank : rank 1, reciprocal rank 1.000, in top 5: True
Before reranking, the correct chunk was outside the context the model would see. After reranking, it is first.
Across many queries, use these metrics:
- Judging the top of the list: Precision@K at your application’s context size shows whether the top results improved.
- Grading the whole order: NDCG reflects graded relevance across the list.
- Finding one known answer: Mean reciprocal rank suits single-answer lookups.
Then judge the end answer with RAG evaluation. A better-ranked list that the generator ignores changes nothing.
Worked example: checking a reranking win
Return to the help-center query. The trace for that request looks like this:
stage 1 retrieval (50 candidates): correct chunk at rank 7
stage 2 rerank (cross-encoder): correct chunk at rank 1
generation context: top 5 after rerank -> answer cites the refund rule
Before you count this as a win, check two things.
First, confirm the correct chunk was already in the candidate set. Here it was at rank 7, so reranking had something to promote. If it had been missing, no reranker could have helped.
Second, confirm the added latency stayed within the page budget. A reranker that fixes the order but slows the page past its limit may not be worth deploying.
For more on that first check, see retrieval quality.
How to debug reranking
When answers don’t improve after you add a reranker, check each stage with its own metric.
- Candidate recall: Was the correct chunk in the candidate set at all? If not, fix retrieval first.
- Rank after reranking: Did the reranker move the correct chunk above the context cutoff?
- Generator use: Did the model actually use the top chunk in its answer?
Log the per-stage rank for every chunk an answer cites. That lets you see where a good chunk was lost.
Reranking vs. better retrieval
Improving chunking, indexing, or the embedding model raises candidate recall. Reranking improves the order of a set you already have.
Use your evaluation results to decide where to spend effort:
- The right chunk is usually missing: Work on retrieval.
- The right chunk is usually present but buried: Reranking is the cheaper fix.
Hybrid fusion and reranking can work together. Fusion merges retrievers by rank, and then a reranker makes the final, query-aware ordering. For more on fusion, see hybrid search and reciprocal rank fusion.
Limitations
Reranking helps the order of results, but it has costs and limits.
- Added latency and cost: Every candidate needs a model call at query time.
- Candidate set size: A set that is too small caps the benefit. A set that is too large uses up the latency budget.
- No recall fix: A reranker cannot recover a chunk that retrieval never returned.
- Inherited bias: Rerankers can carry biases from their training data.
- Possible demotion: A reranker may push down a chunk the generator would have used well.
- Uncalibrated scores: Reranker scores are ordering signals, not probabilities of relevance.
Frequently asked questions
How many candidates should I rerank?
Rerank enough candidates to contain the right chunk reliably, which is often tens rather than thousands. Measure candidate recall at your chosen depth. Then weigh that against the latency of scoring that many pairs.
Can reranking replace hybrid search?
Not exactly. Hybrid search broadens and merges candidate sources, while reranking refines the order of one list. Many systems fuse first and then rerank the merged candidates.
Why did reranking not improve my answers?
Usually the correct chunk was missing from the candidates, the generator ignored the top context, or the gain happened below the context cutoff. Check each stage with its own metric before you tune the reranker.
Is a reranker score a probability?
No. Treat it as a ranking score within one query’s candidate set. Comparing raw scores across queries or models is not meaningful without calibration work.
What is the difference between a retriever and a reranker?
A retriever searches the whole index quickly and returns a candidate list. A reranker reads each candidate together with the query and reorders that short list more carefully.
Putting reranking to work
Before you add a reranker, measure candidate recall. If the right chunk is rarely in the candidate set, a reranker won’t help.
When you deploy one, record the candidate depth, the reranker model, where it sits relative to fusion, and the added latency. Compare Precision@K and the final answer quality on the same queries with and without reranking.
Trace each retrieval and reranking step so you can see a chunk’s rank at every stage. Arize Phoenix can capture those retrieval spans, which makes it easier to tell a retrieval miss from a reranking miss.
