Retrieval Augmented Generation and Dense Passage Retrieval use the context of a question to retrieve relevant passages from a large corpus of documents and extract answers. These models act akin to librarians, using their understanding of your question to retrieve relevant books or documents from the library. Modern stacks combine dense vector retrieval, optional sparse search, reranking, and an LLM reader that grounds answers in returned passages.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Retrieval-based QA separates finding evidence (retrieval) from wording the answer (generation).
- Dense passage retrieval embeds questions and passages into vectors for semantic nearest-neighbor search.
- RAG quality depends on chunking, index freshness, encoder version, and rerankers, not only the LLM.
- Eval requires retrieval metrics (recall@k) and answer metrics (groundedness, exact match) together.
- Trace query, hits, scores, and prompts per request to debug misses and hallucinations.
Pipeline stages
Indexing. Chunk documents, embed passages, store in a vector index with metadata filters.
Query encoding. Embed user question (and conversation context) with the same encoder family.
Retrieval. Fetch top-k by cosine or inner product; optionally blend BM25 sparse scores.
Reranking. Cross-encoder or lightweight model reorders candidates for precision.
Generation. LLM reads retrieved passages under a grounded prompt template and cites or summarizes.
Failure at any stage surfaces as wrong answers even with a strong LLM.
Dense passage retrieval (DPR)
Trains dual encoders so relevant question-passage pairs sit close in embedding space. Inference is fast approximate nearest neighbor search over millions of chunks.
Contrast with sparse retrieval (TF-IDF, BM25) which excels on keyword overlap. Hybrid pipelines often merge both score lists.
RAG versus fine-tune alone
RAG updates knowledge by changing the corpus without retraining weights. Fine-tuning bakes facts into parameters but stale quickly.
Many products combine: RAG for fresh docs plus fine-tuned tone and format.
Evaluation
Retrieval: recall@k, MRR, nDCG on labeled question-passage pairs from your domain.
Generation: exact match, rubric grades, citation accuracy, faithfulness judges comparing answer to retrieved text.
End-to-end task success on support tickets or internal QA golden sets.
Log empty retrieval rate and average similarity scores as leading indicators.
Introductory material in introduction to retrieval augmented generation walks the full stack for practitioners.
Practical guidance in getting started with embeddings cover index design and hybrid retrieval.
Agent systems should trace retrieval and generation together per agent evaluation.
Failure modes
Chunks split entities across boundaries so retrieval misses complete facts.
Stale index after product or policy updates.
Encoder mismatch between indexed passages and live queries.
LLM ignores retrieved context and hallucinates despite good hits.
Over-long contexts truncate tail passages containing the answer.
Chunk overlap strategies
Overlapping chunks reduce boundary misses at the cost of index size. Tune overlap on recall@k for questions that cite entity spans crossing chunk borders.
Citation evals
Score whether generated answers cite the passage that actually contains the supporting sentence, not merely whether the answer text is correct.
Practitioner checklist
Before changing production settings, confirm labels, thresholds, and monitor windows match the definitions used in your last offline eval. Snapshot dashboards when incidents start so postmortems compare apples to apples.
Run slice-level reviews on high-traffic cohorts weekly even when global metrics look flat. Many failures appear first in one locale, product line, or prompt route.
Document model version, featurizer hash, and data window in every incident ticket. Future you needs that context to interpret drift charts.
When metrics disagree (offline vs online, precision vs recall), write down the business cost of each error type before picking a fix. Metrics are proxies; costs are the decision.
Share eval harness links in release notes so support and PM teams know which golden tasks must pass before they announce improvements.
FAQ
Is DPR the same as RAG?
DPR is a retrieval method. RAG is the broader pattern of retrieve-then-generate, which may use DPR or other retrievers.
How big should chunks be?
Tune on your eval set; smaller chunks improve precision, larger chunks preserve context.
Do I need reranking?
Helpful when k is large or corpus is noisy; adds latency tradeoff.
How often should I re-embed the corpus?
When encoder, chunking, or source documents change materially; track retrieval metrics weekly.
Can RAG work without vector databases?
You need some retrieval mechanism; vectors are the common semantic approach at scale.
