Retrieval-augmented generation, or RAG, is an architecture where a system retrieves external context and provides it to a model before generation. The goal is to make outputs more accurate, current, and grounded in source material than the model could produce from weights alone.
Nothing about the model changes. What changes is the prompt. At request time the system searches a corpus you control, selects some passages, and puts them in front of the model with the user’s question. The model answers from that text instead of from memory. That one design choice is why RAG is the default pattern for support assistants, internal search, and any agent that has to cite a source.
Key takeaways
- RAG is an inference-time pattern, not a training technique. You change what the model sees, not what it knows.
- Every RAG system has two paths: an offline ingestion path that chunks, embeds, and indexes documents, and an online query path that retrieves, ranks, assembles a prompt, and generates.
- Quality is bounded by retrieval. If the passage that answers the question never reaches the context window, no prompt fixes it.
- RAG reduces hallucination but does not remove it. A model can still contradict, overstate, or ignore the text you gave it.
- Choosing RAG means taking on a data pipeline: freshness, permissions, chunk boundaries, and index maintenance become your problem.
How a RAG pipeline works
The ingestion path runs before any user shows up. Source documents are parsed, split into retrievable units, converted to embeddings, and written to an index along with metadata such as source, timestamp, and access permissions. How you split those documents is a real design decision with its own tradeoffs, covered under chunking strategy.
The query path runs per request:
- Query construction. The raw question is often rewritten, expanded, or decomposed first, especially in a conversation where “what about the second one” only makes sense with history attached.
- Retrieval. The query hits a vector index, a keyword index, or both. Hybrid search is common because embeddings are good at paraphrase and bad at exact identifiers such as
SKU-4471. - Ranking. A cross-encoder reranker reorders the candidate set. Retrieving 50 and reranking to 5 usually beats retrieving 5 directly.
- Prompt assembly. Surviving chunks go into a template with instructions on how to use them, whether to cite, and what to do when the context falls short.
- Generation. The model writes the answer, and any citation or grounding check happens here or immediately after.
Most teams start with fixed-size chunks, one embedding model, and plain top-k similarity search. That is a fine first pass at a RAG pipeline and it is usually enough to learn which parts of your corpus are actually hard. The reranker, the hybrid index, and the query rewriter get added later, after the failures tell you which one you need.
RAG, fine-tuning, and long context
These get framed as competitors more often than they should be.
Fine-tuning changes behavior: format, tone, task structure, adherence to a schema. It is a poor way to install facts, because updating a fact means retraining and you still cannot cite where the fact came from. RAG changes knowledge, cheaply, and the citation comes free because you know which document you passed in.
Long context windows let you skip retrieval by pasting more in. That works until the corpus outgrows the window, and it gets expensive per request well before that. Attention also thins across a very long prompt, so stuffing 200 pages in often scores worse than selecting the right two. Long context does not remove retrieval, it raises the budget for how sloppy retrieval is allowed to be.
In practice the three combine: fine-tune for format, retrieve for facts, and use the larger window to prune less aggressively.
Where RAG quality comes from
Two things have to go right, and they fail differently.
Retrieval has to surface the passage that contains the answer. This is the ceiling on everything downstream, and it is the subject of retrieval quality as a measurement and retrieval failure as a taxonomy of how it breaks.
Generation has to use that passage faithfully. A model handed correct context can still paraphrase it wrong, blend it with a memorized fact, answer a question the context does not address, or fabricate a citation. Prompt ordering, an explicit instruction to abstain when the context is insufficient, and a required quote before the answer all change adherence measurably, which is why research on the generation stage treats it as a distinct problem from search. Grounding failures are common enough to have dedicated benchmarks, including an open source dataset for RAG hallucination detection.
Because both stages can produce a wrong answer, a single score on the final output tells you that something broke without telling you what. Splitting the measurement by stage is the job of RAG evaluation.
What RAG does not fix
RAG inherits the state of your corpus. If the runbook is three versions out of date, retrieval will find it and the model will repeat it confidently. If two documents disagree, the model picks one, usually without flagging the conflict. If a document should not be visible to this user, the filter has to enforce that at query time, because nothing downstream will.
RAG also does not answer questions that require reasoning across many documents at once. Top-k retrieval returns passages that individually resemble the query, so “summarize every incident with this root cause last quarter” quietly returns the three most similar incidents instead of all of them.
FAQ
Is RAG the same as semantic search?
No, though semantic search is usually a component. Semantic search returns a ranked list of documents for a human to read. RAG takes that list, selects a subset that fits a token budget, and hands it to a model that writes prose. The extra steps are where most RAG-specific failures live: chunk selection, prompt assembly, and whether the model stayed faithful to what it received.
Does RAG stop hallucination?
It reduces it substantially and does not eliminate it. RAG removes the most common cause, which is a model answering from vague or outdated parametric memory. It leaves two open paths: retrieval returns nothing useful and the model answers anyway, or retrieval works and the model misreads it. Instructing the model to say it does not know, and measuring how often it complies, matters more than any retrieval upgrade here.
Should I use RAG or fine-tuning?
Fine-tune when the problem is how the model responds. Use RAG when the problem is what the model knows, especially if that knowledge changes weekly or needs a citation. When unsure, start with RAG: faster to build, easier to update, and the failures are legible because you can read exactly what went into the prompt.
How do I know whether my RAG system is working?
Not from spot-checking answers. Trace each request so you can see the retrieved chunks alongside the final output, then score the retrieval step and the generation step separately. When an answer is wrong, the trace tells you whether the right passage was missing, present but ranked below your cutoff, or present and ignored. Those three findings send you to three different fixes.
