Retrieval failure happens when the system does not fetch the information needed to answer or act correctly. It can be caused by missing documents, bad chunking, weak embeddings, poor query rewriting, filters, stale indexes, permissions, or ranking issues.
Retrieval failure often looks like hallucination at the final answer layer. Trace-level RAG evaluation helps separate “the model ignored evidence” from “the evidence was never retrieved.”
That list of causes is long, but it collapses into four classes, and knowing which class you are in determines the fix. Debugging retrieval without classifying first is how teams end up swapping embedding models to solve an ingestion bug.
Key takeaways
- Four classes cover nearly all of it: the answer is not in the index, it is in the index but ranks too low, it was retrieved but arrived unusable, or the query was wrong.
- Each class has a different fix, and three of the four are not fixed by a better retriever.
- A model ignoring correct context is a generation failure, not a retrieval failure. Keeping that line clean is the point of the taxonomy.
- The symptom is identical across all four classes, so classification happens in the trace, not in the output.
- The class that hurts most is silent: no error, no empty result, just a plausible answer built from the wrong passages.
Class 1: nothing relevant is in the index
Retrieval did its job against a corpus that never contained the answer. Common causes:
- The document was never ingested, or the ingestion job failed silently on one source.
- The parser dropped it. Scanned PDFs, tables, and text inside images regularly produce empty or garbled output that gets embedded anyway.
- The index is stale. The document was updated, the version you indexed was not.
- A permission or metadata filter removed it before scoring. Correct behavior if the user genuinely lacks access, a bug if the filter is over-broad.
This class is invisible to retrieval metrics computed over the corpus you have, since your labels come from that same corpus. It surfaces as a cluster of questions about one subject that are consistently answered badly. The fix is in the data pipeline.
Class 2: it exists but ranks below the cutoff
The passage is in the index and did not make the top k. This is the most common class in a working system and the most tractable.
Vocabulary mismatch is the usual driver. Embeddings match paraphrase well and exact strings badly, so a query for ERR_CONN_4021 competes poorly against passages that discuss connection errors in prose. Adding keyword search alongside vector search recovers most of these.
The second driver is missing reranking. Bi-encoder similarity compares two vectors computed independently, which is fast and coarse. A cross-encoder reads query and passage together and reorders accordingly.
A quick diagnostic: rerun the failing query with k set to 50. If the needed passage appears at rank 23, this is your class, and it is a ranking problem.
Class 3: retrieved but unusable
The right chunk came back and still could not do its job.
- Split mid-answer. The boundary landed between a condition and its exception, so the retrieved text says a refund is available and “except for annual plans” sits in the next chunk. Benchmarking on how chunking choices change RAG results is where to start when this class dominates.
- Stripped of context. A chunk reading “This applies to all customers in the second tier” is unusable when the heading naming the tier is three chunks up.
- Flattened structure. Tables that lost their column headers, code that lost its indentation, lists that lost their ordering.
- Dropped at assembly. Retrieved, ranked well, then trimmed by the token budget because earlier chunks used the space.
Retrieval metrics score this class as a success, since the correct document was returned. Only reading the context actually sent to the model catches it.
Class 4: the query was wrong
Retrieval faithfully answered a question nobody asked.
Conversational fragments are the classic case: “what about the enterprise one” embedded on its own retrieves nothing useful, because the referent is two turns back. Rewriting against conversation history fixes it.
Multi-hop questions fail differently. “Which of our regions is affected by the policy we updated last week” needs one search to identify the policy and another to find the affected regions. A single call returns passages resembling the whole sentence and answers neither part.
Agents add their own version. When retrieval is a tool call, the agent picks the arguments and can pass an over-narrow filter, a date range that excludes the answer, or a rephrasing it prefers. It can also skip retrieval and answer from memory, which is a retrieval failure with no retrieval span in the trace at all.
What is not retrieval failure
If the correct passage reached the model and the answer contradicts it, retrieval succeeded and generation failed. Same symptom, opposite fix.
Making that call mechanically means capturing the retrieved chunks as span attributes on every request. Once a trace records each span with its inputs and outputs, classification is reading two fields: what came back from retrieval, and what the model did with it. Without that, every wrong answer is a guess about which stage to blame, and answers built on absent evidence keep getting filed as model hallucinations when they are search problems.
FAQ
How do I tell retrieval failure from hallucination?
Read the retrieved context for that specific request. Present in the context and contradicted by the answer means generation failure. Absent from the context but present in the corpus means retrieval failure. Absent from both means a coverage gap, and the model should have declined instead of answering. All three produce the same wrong answer, so the trace is the only place the difference is visible.
What is retrieval inefficiency?
The term gets used for two different things. Sometimes it means the quality problem on this page: the system does work and returns the wrong material. Sometimes it means resource cost: retrieving 50 chunks to use 3, re-searching on every conversational turn, or an agent looping on near-identical queries. The second kind shows up as latency and token spend rather than wrong answers, and it is worth tracking separately, since a system can be accurate and wasteful at once.
Why does my RAG system fail on questions that span multiple documents?
Because top-k similarity retrieves passages that individually resemble the query, and a comparison or aggregation question resembles no single passage. “List every incident with this root cause” returns the few most similar incidents and stops. The fixes are structural: decompose the question into several searches, add metadata filters so the query can be scoped rather than matched, or precompute the aggregations you know users will ask for.
Can permissions cause retrieval failure?
Yes, in both directions. An over-broad filter removes documents the user is entitled to see, producing failures that reproduce only for certain users and are easy to miss when testing with an admin account. An under-enforced filter is worse, since retrieval succeeds and surfaces content nobody should have seen. Either way the check belongs at query time, because nothing downstream recovers the mistake.
Does a larger context window make retrieval failure go away?
No, it changes which class you hit. A bigger window reduces class 3 truncation and lets you raise k against class 2 ranking misses. It does nothing for a document that was never indexed or a query that asked the wrong thing.
