Agentic RAG Guide

TL;DR

• Agentic RAG replaces the fixed retrieve-then-generate pipeline with an agent that decides whether to search, what to search for, how to evaluate results, and when the answer is sufficient.

• Vanilla RAG always retrieves once - agentic RAG may retrieve zero times (the LLM already knows), multiple times (multi-hop), or with reformulated queries (when initial results are poor).

• Self-evaluation is the key capability - the agent assesses retrieved context relevance and its own answer quality before responding, triggering additional retrieval or revision if needed.

• Agentic RAG costs more but handles harder questions - multi-hop queries, ambiguous questions, and tasks requiring synthesis across sources benefit most.

• Start with vanilla RAG - add agentic retrieval only when eval shows retrieval failures that reformulation or multi-hop would fix.

Why This Matters

Vanilla RAG works well for straightforward Q&A: "What is our refund policy?" But production questions are rarely that clean:

• "Compare our refund policy with what we told customer X last month" - requires multiple retrievals and synthesis.
• "Why did revenue drop?" - requires iterative search across different document types.
• "Is this claim in the contract accurate?" - requires retrieval, verification, and potentially re-search with refined terms.

Fixed pipelines fail on these because they retrieve once with the raw query, pass whatever comes back to the LLM, and hope for the best. Agentic RAG treats retrieval as a tool the agent wields strategically - searching when needed, reformulating when results are poor, and verifying before answering.

If you're building RAG for complex enterprise knowledge bases where questions aren't simple lookups, agentic patterns are the difference between a demo and a system that handles real user queries.

The Problem Agentic RAG Solves

Vanilla RAG has structural limitations:

1. Single retrieval pass - if the first search misses relevant documents, the answer is wrong. No retry.

2. Query-document mismatch - users ask questions differently than documents are written. Raw query embedding may miss relevant chunks.

3. No retrieval necessity check - the pipeline always searches, even when the LLM could answer from context or general knowledge. Wasted latency and noise.

4. No answer verification - the LLM generates from whatever was retrieved, even if context is irrelevant or insufficient.

5. Multi-hop blindness - "Who is the CEO of the company that acquired DataCorp?" requires finding DataCorp's acquirer first, then the CEO. Single-pass retrieval can't chain.

Agentic RAG addresses each: adaptive retrieval decisions, query reformulation, self-evaluation, and multi-step search chains.

How We Got Here

Agentic RAG sits on top of the RAG stack, adding a control loop around retrieval:

Diagram: From single-shot RAG to agentic retrieval

flowchart LR
    A[Vanilla RAG] --> B[HyDE / multi-query]
    B --> C[Self-RAG / CRAG]
    C --> D[Agent + retrieve tools]
    D --> E[Hybrid + GraphRAG tools]

Major components and how control or data moves between them.

Public foundations: Lewis et al., RAG (2020), Self-RAG, CRAG, and production graphs in LangGraph / LlamaIndex.

What Is Agentic RAG?

Agentic RAG is a pattern where an AI agent controls the retrieval process - deciding if, when, what, and how to search - rather than executing a fixed index → retrieve → generate pipeline.

The agent typically has access to retrieval tools:

@tool
def search_knowledge_base(query: str, filters: dict = None) -> str:
    """Search internal documents. Returns top-k relevant chunks."""

@tool
def search_with_metadata(query: str, doc_type: str, date_range: str) -> str:
    """Filtered search by document type and date range."""

@tool
def get_document(doc_id: str) -> str:
    """Retrieve full document by ID for detailed reading."""

The agent loop decides:

• Retrieve or not? - "Do I need external data to answer this?"

• What query? - Reformulate the user's question for better retrieval.

• Enough context? - Evaluate retrieved chunks for relevance and completeness.

• Answer or retry? - Generate response, or search again with refined terms.

This maps to research concepts:

• Self-RAG - the model generates retrieval decisions and self-critique tokens.

• Corrective RAG (CRAG) - evaluates retrieval quality and triggers web search or query reformulation on failure.

• Adaptive RAG - routes queries to different retrieval strategies based on complexity.

In production, these converge into agent loops with retrieval tools and evaluation steps.

How Agentic RAG Works

The Agentic RAG Loop

Diagram: Agentic RAG retrieval loop

flowchart TD
    Q[User query] --> D{Need retrieval?}
    D -->|No| G[Generate]
    D -->|Yes| F[Formulate query]
    F --> R[Retrieve + rerank]
    R --> E{Relevant + sufficient?}
    E -->|No / under cap| F
    E -->|Yes| G
    G --> V{Answer verified?}
    V -->|No| F
    V -->|Yes| Out[Respond + cite]

Major components and how control or data moves between them.

RAG couples a dense vector index of external knowledge with a generator. Vanilla RAG always retrieves once; agentic RAG makes retrieval a decision inside the loop.

Step 1: Retrieval Decision

Not every query needs RAG. The agent assesses:

Query: "What is Python?"
→ No retrieval needed (general knowledge)

Query: "What is our Q3 headcount?"
→ Retrieval needed (private data)

Query: "Summarize the meeting we discussed earlier"
→ Check conversation memory first, then retrieve if needed

This saves latency on queries the LLM can answer directly and prevents irrelevant context from degrading response quality.

Step 2: Query Formulation

Users don't search like documents read. The agent reformulates:

HyDE (Hypothetical Document Embeddings) is a common technique - the agent generates a hypothetical answer, embeds that, and searches for documents similar to the hypothetical answer rather than the question.

Step 3: Retrieval Evaluation

After retrieving, the agent evaluates context quality before generation:

class RetrievalEvaluation(BaseModel):
    is_relevant: bool
    is_sufficient: bool
    missing_information: str | None
    suggested_requery: str | None

If is_relevant=False → reformulate query or try different source. If is_sufficient=False → additional retrieval for missing information. If both pass → proceed to generation.

Step 4: Answer Verification

After generating, the agent verifies the answer against retrieved context:

• Are all claims supported by sources?
• Are citations accurate?
• Does the answer actually address the question?

Failed verification triggers revision or additional retrieval - the Self-RAG pattern.

Architecture

Agentic RAG extends the classic offline/online RAG stack with control modules:

Diagram: Agentic RAG system architecture

flowchart TB
    subgraph offline [Offline index]
        Load --> Chunk --> Embed --> Store[(Vector + BM25)]
    end
    subgraph online [Online agentic query]
        QD[Retrieval decision] --> QE[Query engine]
        QE --> Ret[Retrieve + rerank]
        Ret --> Ev[Context evaluator]
        Ev -->|retry| QE
        Ev -->|ok| Gen[Generate]
        Gen --> Ver[Answer verifier]
        Ver -->|retry| QE
        Ver -->|ok| Out[Cited answer]
    end
    Store --> Ret

Major components and how control or data moves between them.

Figure: Classic RAG separates offline indexing from online retrieve → generate. Agentic RAG adds decision, reformulation, evaluation, and verification around that online path.
Source: Gao et al., Retrieval-Augmented Generation for Large Language Models: A Survey (arXiv:2312.10997)

Step-by-Step Flow

Query: "How does our data retention policy compare to GDPR requirements?"

1. Retrieval decision - Agent determines: requires internal policy docs + regulatory knowledge. Proceed with retrieval.

2. Query formulation - Agent generates two search queries:

   • "data retention policy internal document"
   • "GDPR data retention requirements minimum periods"

3. First retrieval - Vector search returns 5 chunks from internal policy doc. BM25 search returns GDPR summary chunks.

4. Evaluation - Internal policy: relevant ✓, sufficient ✓. GDPR: relevant ✓, but missing specific retention period requirements. suggested_requery: "GDPR Article 5 storage limitation periods".

5. Second retrieval - Targeted search returns GDPR Article 5 excerpts with specific retention periods.

6. Re-evaluation - Both contexts now sufficient ✓

7. Generation - Agent synthesizes comparison: "Our policy retains customer data for 7 years. GDPR requires retention no longer than necessary for the stated purpose - typically 2-6 years depending on data category. Gap: our retention exceeds GDPR for marketing data."

8. Verification - Each claim mapped to source chunk. Citation check passes ✓

9. Response - Answer delivered with citations to internal policy section 4.2 and GDPR Article 5(1)(e).

Total: 2 retrieval passes, 1 evaluation cycle, 1 verification - vs. vanilla RAG's single pass that likely misses GDPR specifics.

Real Production Example

LangGraph agentic RAG with retrieval tools and evaluation:

from typing import Annotated, Literal
from pydantic import BaseModel, Field
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_core.tools import tool
from langgraph.graph import StateGraph, END, MessagesState
from langgraph.prebuilt import ToolNode

# --- Retrieval tools ---
vectorstore = Chroma(persist_directory="./kb", embedding_function=OpenAIEmbeddings())
reranker = CohereRerank(model="rerank-english-v3.0")

@tool
def search_documents(query: str, top_k: int = 5) -> str:
    """Search the knowledge base for relevant documents."""
    results = vectorstore.similarity_search(query, k=top_k * 2)
    reranked = reranker.rerank(query, results, top_k=top_k)
    return format_chunks(reranked)

@tool
def search_by_metadata(query: str, doc_type: str) -> str:
    """Search with document type filter (policy, contract, faq, technical)."""
    results = vectorstore.similarity_search(
        query, k=5, filter={"doc_type": doc_type}
    )
    return format_chunks(results)

tools = [search_documents, search_by_metadata]
tool_node = ToolNode(tools)

# --- Evaluation ---
class ContextEval(BaseModel):
    relevant: bool
    sufficient: bool
    requery: str | None = None

evaluator = ChatOpenAI(model="gpt-4o-mini", temperature=0)

def evaluate_context(state: MessagesState):
    last_context = extract_last_tool_result(state["messages"])
    eval_result = evaluator.with_structured_output(ContextEval).invoke([
        {"role": "system", "content": "Evaluate if retrieved context can answer the user's question."},
        {"role": "user", "content": f"Question: {get_user_query(state)}\nContext: {last_context}"},
    ])
    if not eval_result.relevant or not eval_result.sufficient:
        return {"needs_retry": True, "requery": eval_result.requery}
    return {"needs_retry": False}

# --- Agent graph ---
llm = ChatOpenAI(model="gpt-4o", temperature=0).bind_tools(tools)

def agent_node(state: MessagesState):
    response = llm.invoke([
        {"role": "system", "content": """You are a research agent with access to a knowledge base. 1. Decide if you need to search (skip for general knowledge questions). 2. Formulate effective search queries (not the user's raw question). 3. Search multiple times if needed for complex questions. 4.

Cite sources in your final answer."""},
        *state["messages"],
    ])
    return {"messages": [response]}

def route(state: MessagesState) -> Literal["tools", "evaluate", "end"]:
    last = state["messages"][-1]
    if last.tool_calls:
        return "tools"
    if state.get("needs_retry"):
        return "agent"
    return "end"

graph = StateGraph(MessagesState)
graph.add_node("agent", agent_node)
graph.add_node("tools", tool_node)
graph.add_node("evaluate", evaluate_context)
graph.set_entry_point("agent")
graph.add_conditional_edges("agent", route, {"tools": "tools", "evaluate": "evaluate", "end": END})
graph.add_edge("tools", "evaluate")
graph.add_conditional_edges("evaluate", lambda s: "agent" if s.get("needs_retry") else "end", {"agent": "agent", "end": END})

agentic_rag = graph.compile()

For simpler cases, a ReAct agent with search tools (create_react_agent) provides agentic retrieval without explicit evaluation nodes - the agent self-evaluates in its thought traces.

Design Decisions

Comparisons

Agentic RAG vs naive RAG

Agentic RAG vs GraphRAG

See GraphRAG and Hybrid Search - agentic control does not replace good indexing.

Common Mistakes

1. Agentic RAG on top of bad retrieval - If your chunks, embeddings, or index are poor, agentic reformulation won't help. Fix vanilla RAG first.

2. Unbounded retrieval loops - Agent keeps searching without converging. Cap at 2–3 retrieval passes with explicit stop conditions.

3. Always retrieving - Agent searches even for "hello" or general knowledge questions. Add explicit retrieval necessity check.

4. No citation enforcement - Agentic RAG synthesizes across multiple retrievals. Without citation requirements, sources become untraceable.

5. Skipping reranking - Agent retrieves 5 chunks × 3 passes = 15 chunks in context. Rerank aggressively to keep context focused.

6. Over-engineering simple Q&A - "What's our office address?" doesn't need agentic RAG. Route by query complexity.

7. Ignoring cost - Each retrieval pass adds embedding + search + evaluation LLM calls. Monitor cost per query type.

Where It Breaks Down

• Real-time data - Agentic retrieval from stale indexes produces confident wrong answers. Freshness metadata and cache invalidation are prerequisites.

• Very large corpora - Reformulation helps but can't fix fundamental recall issues at billion-chunk scale. Need better indexing, not more agent steps.

• Adversarial documents - Retrieved content containing prompt injection affects agent decisions across multiple passes.

Sanitize retrieved text.

• Latency budgets - 3 retrieval passes × (embed + search + rerank + eval) = 3–10s before generation. Unacceptable for some UX patterns.

• Evaluation reliability - LLM-as-judge for context sufficiency inherits model biases. False "sufficient" ratings skip needed retrieval.

Decision tree: RAG vs Agentic RAG vs GraphRAG

Decision tree: Choosing a retrieval pattern

flowchart TD
    A[Need external knowledge?] -->|No| B[Direct generation]
    A -->|Yes| C{Simple lookup / FAQ?}
    C -->|Yes| D[Vanilla RAG]
    C -->|No| E{Multi-hop entity relationships?}
    E -->|Yes| F[GraphRAG / knowledge graph]
    E -->|No| G{First-pass context often incomplete?}
    G -->|Yes| H[Agentic RAG with reformulation]
    G -->|No| D
    H --> I[Cap retrieval passes + cite sources]
    F --> I

Escalate retrieval complexity only when evals show vanilla RAG missing multi-hop or under-specified queries.

When NOT to Use Agentic RAG

Skip agentic retrieval when:

1. Vanilla RAG already hits your eval bar - don't pay 2–4× for marginal gains.
2. Queries are simple lookups - FAQ, policy snippet, "what's the office address?"
3. Latency budgets are tight - multi-pass often means seconds before first token.
4. The index/chunking is broken - fix RAG infrastructure before adding loops.
5. You cannot cite or verify - multi-pass synthesis without citations becomes untraceable.

Route complex queries to agentic RAG; keep a vanilla path for the rest.

Running in Production

Best Practice

✅ Best Practices - Route by query complexity, cap retrieval passes, rerank every pass, require citations, and prove lift on a fixed eval set before widening agentic routing.

Important

Build and evaluate vanilla RAG first. Add agentic retrieval only when your eval set shows failures that multi-pass or reformulation would fix - measure the improvement against the cost increase.

Related Guides

• Models: GPT-5 · Claude Sonnet · Claude Fable · Claude Haiku — agent brains for adaptive retrieval loops.

• Companies: OpenAI · Anthropic — providers for agentic RAG stacks.

• LangGraph: Agentic RAG graphs with evaluation nodes and conditional retry - see Best AI Agent Frameworks.

• LlamaIndex: Query engines with sub-question decomposition and agent query engines.

• Self-RAG / CRAG: Research patterns for self-evaluation and corrective retrieval.

• RAGAS: Evaluation framework extensible to agentic metrics.

• Cohere Rerank / vector DBs: Keep multi-pass context focused - Best Vector Databases.

• Comparisons: Pinecone vs Weaviate · Qdrant vs Pinecone · LangGraph vs CrewAI

• RAG: Foundation pattern - agentic RAG extends it with adaptive retrieval control.

• AI Agents: Agentic RAG is an agent with retrieval tools and evaluation loops.

• Agent Architectures: ReAct and Plan-and-Execute structure the agentic retrieval loop.

• GraphRAG: Graph-based retrieval for multi-hop queries - complementary to agentic search.

• Hybrid Search: Infrastructure layer that agentic RAG should use, not replace.

• Retrieval Evaluation: Eval metrics extend to multi-pass retrieval scenarios.

• Agent Memory: Conversation context informs retrieval decisions across turns.

• Guardrails: Faithfulness and citation rails pair with answer verification.

If you understood this topic, read next:

Diagram: Learning path for retrieval agents

flowchart LR
    A[RAG] --> B[Agents]
    B --> C[Agentic RAG]
    C --> D[GraphRAG]
    D --> E[Retrieval eval]
    E --> F[Memory]

Prerequisites: RAG · AI Agents · Agent Architectures

Next topics: GraphRAG · Retrieval Evaluation · Agent Memory

Estimated time: 55 min · Difficulty: Advanced

Key Takeaways

• Agentic RAG gives an agent control over retrieval - when to search, what to query, how to evaluate, and when to retry.
• It solves multi-hop queries, query-document mismatch, and insufficient context that vanilla RAG cannot recover from.
• Self-evaluation (context relevance + answer faithfulness) is the defining capability - not just multi-pass search.
• Cap retrieval retries (2–3 max) and route simple queries to vanilla RAG to control cost.
• Fix chunking, hybrid search, and reranking before adding agentic complexity.
• GraphRAG is complementary: use graph tools inside an agentic loop when relationships matter.
• Evaluate with standard RAG metrics plus passes per query, reformulation success, and routing accuracy.

FAQs

When should I use agentic RAG vs. vanilla RAG?

Vanilla RAG for straightforward Q&A with good retrieval recall. Agentic RAG when queries require multi-hop search, query reformulation, comparison across documents, or answer verification.

Is agentic RAG the same as Self-RAG?

Self-RAG is a specific research approach where the model generates special tokens for retrieval decisions and self-critique. Agentic RAG is the broader production pattern - agents with retrieval tools and evaluation loops. Self-RAG is one implementation.

How many retrieval passes are normal?

1–3 for most queries. If you're consistently hitting 3+, your indexing or query formulation needs improvement - not more retrieval capacity.

Does agentic RAG replace reranking?

No. Reranking is more important with agentic RAG because multiple passes accumulate chunks. Rerank at each pass to keep context focused.

Can I use agentic RAG with GraphRAG?

Yes. An agent might use vector search for initial retrieval and graph traversal for relationship queries - choosing the tool based on query type.

How do I route between vanilla and agentic RAG?

Query classifier (LLM or rules): simple factual lookups → vanilla pipeline; complex analytical questions → agentic pipeline. Measure routing accuracy in eval.

What's Corrective RAG (CRAG)?

CRAG evaluates retrieval quality with a scorer. Low scores trigger query reformulation or fallback to web search. It's a specific agentic pattern focused on retrieval quality gates.

How do I evaluate agentic RAG?

Standard RAG metrics (faithfulness, relevance) plus: retrieval necessity accuracy, average passes per query, reformulation success rate, and multi-hop task completion rate.

Is agentic RAG worth the cost?

For enterprise Q&A with complex queries, typically yes - 2–4× cost for significantly better accuracy on hard questions. For FAQ bots with simple lookups, no.

Can agentic RAG work with local models?

Yes, but evaluation and reformulation steps need capable models. Use a strong local model (Llama 3 70B) or route evaluation to an API model while running retrieval locally.

How does agentic RAG handle conversation context?

Prior turns inform retrieval decisions - "the policy we discussed" resolves to a specific document. Combine with Agent Memory for cross-turn retrieval context.

What's the simplest agentic RAG implementation?

A ReAct agent with a search tool: create_react_agent(llm, [search_documents]). The agent naturally decides when to search and can retry - no explicit evaluation nodes needed.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
• Self-RAG (Asai et al., 2023)
• Corrective Retrieval Augmented Generation (Yan et al., 2024)
• Retrieval-Augmented Generation for Large Language Models: A Survey (Gao et al.)

Further Reading

• LangGraph Documentation
• LlamaIndex RAG Documentation
• RAGAS Evaluation Framework
• Cohere Rerank Documentation