Agentic RAG is retrieval augmented generation where an agent decides how retrieval happens instead of following a fixed retrieve-then-generate path. The agent chooses whether to search at all, what query to issue, which source to hit, whether the results answer the question, and whether to go back and try again.
Classic RAG is a pipeline with one shot at getting retrieval right. Embed the question, pull the top k chunks, stuff them into the prompt, generate. If the question was phrased differently from the documents, or the answer requires combining two sources, or the right chunk ranked eleventh, the pipeline has no recourse. Agentic RAG closes that loop: retrieval becomes a tool the agent calls, evaluates, and calls again.
That flexibility is real, and so is the cost. A single retrieval call becomes an unbounded number of them, and the failure modes multiply along with the steps.
Key takeaways
- Agentic RAG replaces a fixed retrieve-then-generate pipeline with a control loop where the agent decides when and what to retrieve, and whether the results are sufficient.
- The common patterns are query rewriting, routing across multiple sources, self-assessment of retrieved evidence, and iterative retrieval until a stopping condition is met.
- It handles multi-hop questions, ambiguous phrasing, and research-style tasks that single-pass RAG cannot.
- The tradeoffs are cost, latency, and non-determinism. A loop with no hard step budget can quietly spend many times what a single pass would.
- Evaluating agentic RAG requires looking at the trajectory, not just the final answer, because a correct answer can come out of a wasteful or lucky retrieval path.
What the agent actually decides
Whether to retrieve. Plenty of turns do not need retrieval. “Summarize what you just told me” does not warrant a vector search. A single-pass pipeline retrieves anyway, wasting money and injecting irrelevant context that can pull the answer off course.
What to query. Users do not phrase questions the way documents are written. The agent can rewrite “why did my card get declined again” into terms that match the payment failure documentation, split a compound question into separate searches, or add a filter it inferred from earlier in the session.
Where to look. Retrieval is rarely one index. Product documentation, a ticket history, a SQL database, and a web search are different sources with different freshness and different trust levels. Routing is a decision, and getting it wrong looks exactly like a retrieval failure.
Whether the results are good enough. This is the step that distinguishes agentic RAG most sharply. The agent inspects what came back and judges sufficiency before generating. That judgment is itself an evaluation problem, overlapping with the techniques in using an LLM as a judge.
Whether to retrieve again. If evidence is thin or contradictory, the agent reformulates and searches again. Multi-hop questions require this: finding which vendor handles a region, then finding that vendor’s SLA, cannot be done in one lookup.
What stays the same as classic RAG
Agentic RAG does not repair the retrieval layer. Chunking, embeddings, and similarity ranking still determine whether the right passage is reachable. Fix non-agentic retrieval before adding a control loop.
Where agentic RAG fails
Cost and latency multiply silently. Each retrieval iteration is at least one model call to decide, one search, and one call to assess. Three iterations on a hard question is reasonable. Eleven iterations on a question with no answer in the corpus is a bill nobody budgeted. Loops need explicit step budgets and a stopping rule for “the answer is not here.” The second is harder than it sounds, because models are reluctant to conclude that.
Bad query reformulation drifts off target. Rewriting is generative, so it can invent constraints the user never stated. Each rewrite compounds on the last, and by iteration four the agent may be searching for something adjacent to the original question. This is invisible unless intermediate queries are recorded.
Self-assessment is unreliable. Models tend to say retrieved context is sufficient even when it is not.
Sufficient evidence, wrong synthesis. Grounding must be checked separately from retrieval.
Wider attack surface. More retrieval steps mean more ways for untrusted document text to influence behavior.
Evaluating agentic RAG
Final-answer scoring is not enough, because the same answer can come from a clean two-step path or from nine flailing searches. Both look identical at the output. The useful unit of measurement is the trajectory: did the agent retrieve when it should have, were the queries reasonable, was the evidence relevant, was the stopping decision correct, and is the final answer supported by what was actually retrieved.
That decomposition needs the intermediate steps to be recorded as spans, which is the practice described in agent tracing and evaluation. Retry policy, tool definitions, and context assembly live in the harness rather than in the model, a distinction laid out in what an agent harness is and how to evaluate one. Aggregate benchmark numbers deserve skepticism here.
FAQ
What is the difference between RAG and agentic RAG?
Classic RAG runs a fixed sequence: retrieve once, then generate. Agentic RAG puts retrieval inside a control loop where the agent decides whether to search, what to search for, whether the results suffice, and whether to try again. Same components, different control flow, and a much wider range of outcomes and costs.
When is agentic RAG worth the added complexity?
When questions require multiple hops, when the corpus spans sources that need routing, when user phrasing diverges from document phrasing, or when the task is research rather than lookup. For a well-scoped FAQ over one clean corpus, single-pass RAG is cheaper, faster, and easier to debug. Complexity should be added because a measured failure mode demands it.
Does agentic RAG need memory?
Not necessarily, though the two combine well. Retrieval loops benefit from remembering which queries already failed within a session, and longer-running agents benefit from persistent memory across sessions, which is what agentic memory approaches such as A-MEM address. Without at least session-scoped state, an agent will happily reissue a query that returned nothing two steps ago.
How do I stop an agentic RAG loop from running away?
Set a hard maximum on retrieval iterations, cap total tokens per request, and require the agent to state what new information each additional search is meant to find. Then monitor the distribution of iteration counts rather than the average, because the runaway cases live in the tail. Training-time work on multi-turn agent behavior, such as RAGEN, is a different lever on the same underlying difficulty: an agent whose own outputs become its next inputs is hard to keep on a leash.
What should I instrument first?
The retrieval calls. Record the query that was actually issued, which source it hit, what came back, and how the agent scored sufficiency. Most agentic RAG debugging ends at one of those four fields, and none are recoverable after the fact if they were not captured. Getting all four onto the span is the instrumentation baseline.
