What is agentic RAG? Retrieval inside an agent loop
Agentic RAG puts retrieval inside an agent loop: the model decides when to search, grades results and rewrites queries. How it works, with a Python demo.

Agentic RAG is retrieval-augmented generation where an LLM agent controls the retrieval: it decides whether to search, which tool or index to query, whether the results are good enough, and whether to rewrite the query and search again before it answers. Classic RAG runs one fixed retrieve-then-generate pass; agentic RAG turns retrieval into a tool the model calls inside a loop, so it can recover from a bad first search and handle questions that need several hops.
How is agentic RAG different from regular RAG?
In the pipeline I describe in what is RAG, the code decides everything. Embed the question, fetch the top k chunks, paste them into the prompt, generate. The model never sees the search and cannot ask for another one. If the first retrieval misses, the answer is built on the wrong passages anyway.
Agentic RAG hands those decisions to the model. Anthropic’s Building effective agents (19 December 2024) gives the vocabulary I use for this. Its building block is “an LLM enhanced with augmentations such as retrieval, tools, and memory”, and it separates workflows, “systems where LLMs and tools are orchestrated through predefined code paths”, from agents, where “LLMs dynamically direct their own processes and tool usage”. Classic RAG is a workflow. Agentic RAG is retrieval used by an agent. I go deeper on that split in workflows vs agents.
The agentic RAG survey by Singh et al. (15 January 2025) frames it as “embedding autonomous AI agents into the RAG pipeline” and lists the patterns those agents use: “reflection, planning, tool use, and multi-agent collaboration”.
How does agentic RAG work?
Most implementations come down to five moves, which the model makes through tool calls:
- Decide whether to retrieve. Some questions need no lookup. Some need the web, some need the internal wiki, some need a SQL query.
- Plan the query. Rewrite the user’s words into search terms, or split a compound question into sub-questions.
- Retrieve. Call one or more search tools.
- Grade. Judge whether the results actually answer the question.
- Loop or answer. If the evidence is weak, rewrite and search again; if it’s good, answer with citations; if the budget runs out, say so.
Two research lines made steps 1 and 4 concrete before “agentic” was the word for it. Self-RAG (Asai et al., 17 October 2023) trains a model that “adaptively retrieves passages on-demand” and critiques the passages and its own output with “special tokens, called reflection tokens”. Corrective RAG (Yan et al., 29 January 2024) adds a lightweight retrieval evaluator that scores the retrieved documents, uses that confidence to pick a retrieval action, and falls back to web search when the static corpus returns poor documents.
Deep research products are the far end of the same idea. Anthropic’s write-up of its multi-agent research system (13 June 2025) contrasts itself with static RAG directly: instead of fetching the chunks most similar to the query, “our architecture uses a multi-step search that dynamically finds relevant information, adapts to new findings”. I cover that architecture in how deep research agents work.
What does agentic RAG look like in code?
This script compares one-shot RAG with a small retrieve, grade, rewrite loop. The user asks how to rotate an API key; the docs say “regenerate” and “credential” instead. In a real system the grader and the rewriter are LLM calls. Here they are rules, so the script runs with no model and no key, and the control flow is the part to read.
# agentic_rag.py: one-shot RAG vs a retrieve, grade, rewrite loop. Python 3, stdlib only.
# grade() and rewrite() are rules standing in for the LLM calls a real agent would make.
import re
docs = {
"rate-limits": "API keys are limited to 600 requests per minute. Exceeding the API limit returns 429.",
"regenerate": "To replace a credential, open Settings, Tokens, and click Regenerate. The old token stops working after 24 hours.",
"sso": "Single sign-on uses SAML. Admins upload the identity provider metadata in Settings.",
"webhooks": "Webhooks retry five times with exponential backoff before they are marked failed.",
}
SYNONYMS = {"rotate": ["regenerate", "replace"], "key": ["credential", "token"]}
STOP = {"how", "do", "i", "the", "an", "a", "to", "my", "is", "are", "of", "in"}
def words(text):
return {w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP}
def retrieve(query, k=1):
q = words(query)
ranked = sorted(docs, key=lambda d: (-len(q & words(docs[d])), d))
return ranked[:k]
def grade(question, doc_id):
"""Does the passage cover the question's action word? (an LLM judges this in practice)"""
action = words(question) & set(SYNONYMS) - {"key"}
text = words(docs[doc_id])
return all(text & ({a} | set(SYNONYMS[a])) for a in action)
def rewrite(query):
extra = [s for w in sorted(words(query)) for s in SYNONYMS.get(w, [])]
return query + " " + " ".join(extra)
question = "How do I rotate my API key?"
print("One-shot RAG")
hit = retrieve(question)[0]
print(f" retrieved {hit!r}, answer is built from it whether it fits or not\n")
print("Agentic RAG (max 3 steps)")
query = question
for step in range(1, 4):
hit = retrieve(query)[0]
ok = grade(question, hit)
print(f" step {step}: search {query!r}\n -> {hit!r}, relevant={ok}")
if ok:
print(f" answer from [{hit}]: {docs[hit]}")
break
query = rewrite(query)
else:
print(" gave up: say 'not found' instead of guessing")
Output (Python 3.13.11):
One-shot RAG
retrieved 'rate-limits', answer is built from it whether it fits or not
Agentic RAG (max 3 steps)
step 1: search 'How do I rotate my API key?'
-> 'rate-limits', relevant=False
step 2: search 'How do I rotate my API key? credential token regenerate replace'
-> 'regenerate', relevant=True
answer from [regenerate]: To replace a credential, open Settings, Tokens, and click Regenerate. The old token stops working after 24 hours.
The one-shot pipeline matched “API” and “key” to the rate-limits page and would have answered a question about rotation with a paragraph about 429 errors. The loop caught the miss at the grading step, rewrote the query with the vocabulary the docs actually use, and found the right page on the second try. The for ... else branch matters as much as the success path: an agent with a step budget needs an honest way to say it found nothing.
Three things in this toy carry over to production. The grader checks against the original question, not the rewritten query, so rewrites cannot drift the goal. The loop has a hard step limit. And the answer cites the document ID it came from.
What does agentic RAG cost?
More calls, more tokens, more latency. Every grade and rewrite is a model call, and every extra search adds retrieved text to the context. Anthropic’s research-system post puts numbers on the general pattern: agents “typically use about 4× more tokens than chat interactions”, and “multi-agent systems use about 15× more tokens than chats”. It also reports that “token usage by itself explains 80% of the variance” in performance on the BrowseComp evaluation. Spending more tokens is a large part of why these systems work, and it is also the bill.
So the economics favor agentic RAG when a wrong answer is expensive and questions are hard: multi-hop research, ambiguous enterprise search across several systems, anything where a first-pass miss is common. For a high-volume FAQ bot where one hybrid search almost always finds the page, the loop is cost without benefit. Model routing helps here: a small model can do the grading and rewriting while a larger one writes the final answer.
Where does agentic RAG break?
Loops that never converge. A model that keeps rewriting and re-searching burns budget. Cap steps and tokens, and log every tool call.
Graders that are too generous. If the relevance check passes everything, you have one-shot RAG with extra latency. Evaluate the grader on its own with labeled examples; how to evaluate RAG covers the retrieval metrics for that.
Prompt injection through retrieved text. An agent that reads documents and then decides what to do next can be steered by instructions planted in those documents. I wrote about that failure in AI browsers and prompt injection. Keep retrieval tools read-only, and keep any write action behind a confirmation.
Debugging gets harder. A fixed pipeline fails the same way every time. A loop can take a different path on each run. Trace IDs on each step and a replayable log are not optional.
What I would do
I’d start with a fixed pipeline: hybrid search, a reranker, and an eval set. Then I’d look at the failures. If most of them are “the first query used the wrong words” or “the answer needs two lookups”, that is the evidence for adding a loop, and I’d add the smallest one: one grade step and at most two rewrites.
rSearch, the search product I founded, pairs a web search API (Serper) for retrieval with a reasoning model, DeepSeek R1. That is a single retrieval step by design. A loop on top of it would be the agentic version, and the question I’d ask before building it is the same one I’d ask anywhere: what fraction of failures would a second search fix, and what does a second search cost per query?
For multi-system enterprise search, I’d expose each source (wiki, ticket system, SQL warehouse via text-to-SQL) as a separate tool with a clear description, and let the model choose. That is where agentic RAG earns its tokens.
FAQ
What is the difference between RAG and agentic RAG?
RAG runs a fixed sequence: retrieve once, then generate. Agentic RAG lets the model decide when and what to retrieve, check the results and search again before answering.
Is agentic RAG the same as an AI agent?
It is an AI agent whose main tools are retrieval tools. The agent loop is the same one described in what is an AI agent: reason, call a tool, observe, repeat.
Do I need a framework to build agentic RAG?
No. A loop, a search function exposed as a tool, and a step limit are enough, as the script above shows. Anthropic’s agents post also suggests starting with LLM APIs directly rather than a framework.
Is agentic RAG slower than RAG?
Usually, yes. Each grading or rewriting step is another model call and possibly another search, so latency and token use grow with the number of steps.
Sources
- Anthropic, Building effective agents, 19 December 2024
- Anthropic, How we built our multi-agent research system, 13 June 2025
- Singh et al., Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG, arXiv, 15 January 2025
- Asai et al., Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, arXiv, 17 October 2023
- Yan et al., Corrective Retrieval Augmented Generation, arXiv, 29 January 2024