Agentic RAG Guide

TL;DR

• Agentic RAG replaces the fixed retrieve-then-generate pipeline with an agent that decides whether to search, what to search for, how to evaluate results, and when the answer is sufficient.

• Vanilla RAG always retrieves once - agentic RAG may retrieve zero times (the LLM already knows), multiple times (multi-hop), or with reformulated queries (when initial results are poor).

• Self-evaluation is the key capability - the agent assesses retrieved context relevance and its own answer quality before responding, triggering additional retrieval or revision if needed.

• Agentic RAG costs more but handles harder questions - multi-hop queries, ambiguous questions, and tasks requiring synthesis across sources benefit most.

• Start with vanilla RAG - add agentic retrieval only when eval shows retrieval failures that reformulation or multi-hop would fix.

Why This Matters

Vanilla RAG works well for straightforward Q&A: "What is our refund policy?" But production questions are rarely that clean:

• "Compare our refund policy with what we told customer X last month" - requires multiple retrievals and synthesis.
• "Why did revenue drop?" - requires iterative search across different document types.
• "Is this claim in the contract accurate?" - requires retrieval, verification, and potentially re-search with refined terms.

Fixed pipelines fail on these because they retrieve once with the raw query, pass whatever comes back to the LLM, and hope for the best. Agentic RAG treats retrieval as a tool the agent wields strategically - searching when needed, reformulating when results are poor, and verifying before answering.

If you're building RAG for complex enterprise knowledge bases where questions aren't simple lookups, agentic patterns are the difference between a demo and a system that handles real user queries.

The Problem Agentic RAG Solves

Vanilla RAG has structural limitations:

1. Single retrieval pass - if the first search misses relevant documents, the answer is wrong. No retry.

2. Query-document mismatch - users ask questions differently than documents are written. Raw query embedding may miss relevant chunks.

3. No retrieval necessity check - the pipeline always searches, even when the LLM could answer from context or general knowledge. Wasted latency and noise.

4. No answer verification - the LLM generates from whatever was retrieved, even if context is irrelevant or insufficient.

5. Multi-hop blindness - "Who is the CEO of the company that acquired DataCorp?" requires finding DataCorp's acquirer first, then the CEO. Single-pass retrieval can't chain.

Agentic RAG addresses each: adaptive retrieval decisions, query reformulation, self-evaluation, and multi-step search chains.

What Is Agentic RAG?

Agentic RAG is a pattern where an AI agent controls the retrieval process - deciding if, when, what, and how to search - rather than executing a fixed index → retrieve → generate pipeline.

The agent typically has access to retrieval tools:

@tool
def search_knowledge_base(query: str, filters: dict = None) -> str:
    """Search internal documents. Returns top-k relevant chunks."""

@tool
def search_with_metadata(query: str, doc_type: str, date_range: str) -> str:
    """Filtered search by document type and date range."""

@tool
def get_document(doc_id: str) -> str:
    """Retrieve full document by ID for detailed reading."""

The agent loop decides:

• Retrieve or not? - "Do I need external data to answer this?"

• What query? - Reformulate the user's question for better retrieval.

• Enough context? - Evaluate retrieved chunks for relevance and completeness.

• Answer or retry? - Generate response, or search again with refined terms.

This maps to research concepts:

• Self-RAG - the model generates retrieval decisions and self-critique tokens.

• Corrective RAG (CRAG) - evaluates retrieval quality and triggers web search or query reformulation on failure.

• Adaptive RAG - routes queries to different retrieval strategies based on complexity.

In production, these converge into agent loops with retrieval tools and evaluation steps.

How Agentic RAG Works

The Agentic RAG Loop

RAG couples a dense vector index of external knowledge with a sequence-to-sequence generator. At query time, the retriever selects relevant passages and the generator conditions its answer on that evidence.

Step 1: Retrieval Decision

Not every query needs RAG. The agent assesses:

Query: "What is Python?"
→ No retrieval needed (general knowledge)

Query: "What is our Q3 headcount?"
→ Retrieval needed (private data)

Query: "Summarize the meeting we discussed earlier"
→ Check conversation memory first, then retrieve if needed

This saves latency on queries the LLM can answer directly and prevents irrelevant context from degrading response quality.

Step 2: Query Formulation

Users don't search like documents read. The agent reformulates:

HyDE (Hypothetical Document Embeddings) is a common technique - the agent generates a hypothetical answer, embeds that, and searches for documents similar to the hypothetical answer rather than the question.

Step 3: Retrieval Evaluation

After retrieving, the agent evaluates context quality before generation:

class RetrievalEvaluation(BaseModel):
    is_relevant: bool
    is_sufficient: bool
    missing_information: str | None
    suggested_requery: str | None

If is_relevant=False → reformulate query or try different source. If is_sufficient=False → additional retrieval for missing information. If both pass → proceed to generation.

Step 4: Answer Verification

After generating, the agent verifies the answer against retrieved context:

• Are all claims supported by sources?
• Are citations accurate?
• Does the answer actually address the question?

Failed verification triggers revision or additional retrieval - the Self-RAG pattern.

Architecture

Agentic RAG system architecture:

A production RAG stack separates offline indexing (load, chunk, embed, store) from online querying (embed query, retrieve, rerank, generate).

Source: Survey on RAG

Components beyond vanilla RAG:

• Retrieval Decision module - agent or classifier determining search necessity.

• Query Engine - reformulation, HyDE, multi-query generation.

• Evaluator - relevance and sufficiency scoring of retrieved context.

• Answer Verifier - faithfulness check before response delivery.

Step-by-Step Flow

Query: "How does our data retention policy compare to GDPR requirements?"

1. Retrieval decision - Agent determines: requires internal policy docs + regulatory knowledge. Proceed with retrieval.

2. Query formulation - Agent generates two search queries:

   • "data retention policy internal document"
   • "GDPR data retention requirements minimum periods"

3. First retrieval - Vector search returns 5 chunks from internal policy doc. BM25 search returns GDPR summary chunks.

4. Evaluation - Internal policy: relevant ✓, sufficient ✓. GDPR: relevant ✓, but missing specific retention period requirements. suggested_requery: "GDPR Article 5 storage limitation periods".

5. Second retrieval - Targeted search returns GDPR Article 5 excerpts with specific retention periods.

6. Re-evaluation - Both contexts now sufficient ✓

7. Generation - Agent synthesizes comparison: "Our policy retains customer data for 7 years. GDPR requires retention no longer than necessary for the stated purpose - typically 2-6 years depending on data category. Gap: our retention exceeds GDPR for marketing data."

8. Verification - Each claim mapped to source chunk. Citation check passes ✓

9. Response - Answer delivered with citations to internal policy section 4.2 and GDPR Article 5(1)(e).

Total: 2 retrieval passes, 1 evaluation cycle, 1 verification - vs. vanilla RAG's single pass that likely misses GDPR specifics.

Real Production Example

LangGraph agentic RAG with retrieval tools and evaluation:

from typing import Annotated, Literal
from pydantic import BaseModel, Field
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_core.tools import tool
from langgraph.graph import StateGraph, END, MessagesState
from langgraph.prebuilt import ToolNode

# --- Retrieval tools ---
vectorstore = Chroma(persist_directory="./kb", embedding_function=OpenAIEmbeddings())
reranker = CohereRerank(model="rerank-english-v3.0")

@tool
def search_documents(query: str, top_k: int = 5) -> str:
    """Search the knowledge base for relevant documents."""
    results = vectorstore.similarity_search(query, k=top_k * 2)
    reranked = reranker.rerank(query, results, top_k=top_k)
    return format_chunks(reranked)

@tool
def search_by_metadata(query: str, doc_type: str) -> str:
    """Search with document type filter (policy, contract, faq, technical)."""
    results = vectorstore.similarity_search(
        query, k=5, filter={"doc_type": doc_type}
    )
    return format_chunks(results)

tools = [search_documents, search_by_metadata]
tool_node = ToolNode(tools)

# --- Evaluation ---
class ContextEval(BaseModel):
    relevant: bool
    sufficient: bool
    requery: str | None = None

evaluator = ChatOpenAI(model="gpt-4o-mini", temperature=0)

def evaluate_context(state: MessagesState):
    last_context = extract_last_tool_result(state["messages"])
    eval_result = evaluator.with_structured_output(ContextEval).invoke([
        {"role": "system", "content": "Evaluate if retrieved context can answer the user's question."},
        {"role": "user", "content": f"Question: {get_user_query(state)}\nContext: {last_context}"},
    ])
    if not eval_result.relevant or not eval_result.sufficient:
        return {"needs_retry": True, "requery": eval_result.requery}
    return {"needs_retry": False}

# --- Agent graph ---
llm = ChatOpenAI(model="gpt-4o", temperature=0).bind_tools(tools)

def agent_node(state: MessagesState):
    response = llm.invoke([
        {"role": "system", "content": """You are a research agent with access to a knowledge base. 1. Decide if you need to search (skip for general knowledge questions). 2. Formulate effective search queries (not the user's raw question). 3. Search multiple times if needed for complex questions. 4.

Cite sources in your final answer."""},
        *state["messages"],
    ])
    return {"messages": [response]}

def route(state: MessagesState) -> Literal["tools", "evaluate", "end"]:
    last = state["messages"][-1]
    if last.tool_calls:
        return "tools"
    if state.get("needs_retry"):
        return "agent"
    return "end"

graph = StateGraph(MessagesState)
graph.add_node("agent", agent_node)
graph.add_node("tools", tool_node)
graph.add_node("evaluate", evaluate_context)
graph.set_entry_point("agent")
graph.add_conditional_edges("agent", route, {"tools": "tools", "evaluate": "evaluate", "end": END})
graph.add_edge("tools", "evaluate")
graph.add_conditional_edges("evaluate", lambda s: "agent" if s.get("needs_retry") else "end", {"agent": "agent", "end": END})

agentic_rag = graph.compile()

For simpler cases, a ReAct agent with search tools (create_react_agent) provides agentic retrieval without explicit evaluation nodes - the agent self-evaluates in its thought traces.

Design Decisions

⚠ Common Mistakes

1. Agentic RAG on top of bad retrieval - If your chunks, embeddings, or index are poor, agentic reformulation won't help. Fix vanilla RAG first.

2. Unbounded retrieval loops - Agent keeps searching without converging. Cap at 2–3 retrieval passes with explicit stop conditions.

3. Always retrieving - Agent searches even for "hello" or general knowledge questions. Add explicit retrieval necessity check.

4. No citation enforcement - Agentic RAG synthesizes across multiple retrievals. Without citation requirements, sources become untraceable.

5. Skipping reranking - Agent retrieves 5 chunks × 3 passes = 15 chunks in context. Rerank aggressively to keep context focused.

6. Over-engineering simple Q&A - "What's our office address?" doesn't need agentic RAG. Route by query complexity.

7. Ignoring cost - Each retrieval pass adds embedding + search + evaluation LLM calls. Monitor cost per query type.

Where It Breaks Down

• Real-time data - Agentic retrieval from stale indexes produces confident wrong answers. Freshness metadata and cache invalidation are prerequisites.

• Very large corpora - Reformulation helps but can't fix fundamental recall issues at billion-chunk scale. Need better indexing, not more agent steps.

• Adversarial documents - Retrieved content containing prompt injection affects agent decisions across multiple passes.

Sanitize retrieved text.

• Latency budgets - 3 retrieval passes × (embed + search + rerank + eval) = 3–10s before generation. Unacceptable for some UX patterns.

• Evaluation reliability - LLM-as-judge for context sufficiency inherits model biases. False "sufficient" ratings skip needed retrieval.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Important

Build and evaluate vanilla RAG first. Add agentic retrieval only when your eval set shows failures that multi-pass or reformulation would fix - measure the improvement against the cost increase.

Ecosystem

• LangGraph: Agentic RAG graphs with evaluation nodes, conditional retry, and tool-based retrieval.

• LlamaIndex: QueryEngine with sub-question decomposition, recursive retrieval, and agent query engines.

• Self-RAG / CRAG: Research implementations for self-evaluation and corrective retrieval.

• RAGAS: Evaluation framework extensible to agentic metrics (retrieval decision accuracy, faithfulness across passes).

• Cohere: Reranking models critical for keeping multi-pass retrieval context focused.

Related Technologies

• RAG: Foundation pattern - agentic RAG extends it with adaptive retrieval control.

• AI Agents: Agentic RAG is an agent with retrieval tools and evaluation loops.

• Agent Architectures: ReAct and Plan-and-Execute structure the agentic retrieval loop.

• GraphRAG: Graph-based retrieval for multi-hop queries - complementary to agentic search.

• Hybrid Search: Infrastructure layer that agentic RAG should use, not replace.

• Retrieval Evaluation: Eval metrics extend to multi-pass retrieval scenarios.

• Agent Memory: Conversation context informs retrieval decisions across turns.

Learning Path

Prerequisites: RAG · AI Agents · Agent Architectures

Next topics: GraphRAG · Retrieval Evaluation · Agent Memory

Estimated time: 55 min · Difficulty: Advanced

FAQs

When should I use agentic RAG vs. vanilla RAG?

Vanilla RAG for straightforward Q&A with good retrieval recall. Agentic RAG when queries require multi-hop search, query reformulation, comparison across documents, or answer verification.

Is agentic RAG the same as Self-RAG?

Self-RAG is a specific research approach where the model generates special tokens for retrieval decisions and self-critique. Agentic RAG is the broader production pattern - agents with retrieval tools and evaluation loops. Self-RAG is one implementation.

How many retrieval passes are normal?

1–3 for most queries. If you're consistently hitting 3+, your indexing or query formulation needs improvement - not more retrieval capacity.

Does agentic RAG replace reranking?

No. Reranking is more important with agentic RAG because multiple passes accumulate chunks. Rerank at each pass to keep context focused.

Can I use agentic RAG with GraphRAG?

Yes. An agent might use vector search for initial retrieval and graph traversal for relationship queries - choosing the tool based on query type.

How do I route between vanilla and agentic RAG?

Query classifier (LLM or rules): simple factual lookups → vanilla pipeline; complex analytical questions → agentic pipeline. Measure routing accuracy in eval.

What's Corrective RAG (CRAG)?

CRAG evaluates retrieval quality with a scorer. Low scores trigger query reformulation or fallback to web search. It's a specific agentic pattern focused on retrieval quality gates.

How do I evaluate agentic RAG?

Standard RAG metrics (faithfulness, relevance) plus: retrieval necessity accuracy, average passes per query, reformulation success rate, and multi-hop task completion rate.

Is agentic RAG worth the cost?

For enterprise Q&A with complex queries, typically yes - 2–4× cost for significantly better accuracy on hard questions. For FAQ bots with simple lookups, no.

Can agentic RAG work with local models?

Yes, but evaluation and reformulation steps need capable models. Use a strong local model (Llama 3 70B) or route evaluation to an API model while running retrieval locally.

How does agentic RAG handle conversation context?

Prior turns inform retrieval decisions - "the policy we discussed" resolves to a specific document. Combine with Agent Memory for cross-turn retrieval context.

What's the simplest agentic RAG implementation?

A ReAct agent with a search tool: create_react_agent(llm, [search_documents]). The agent naturally decides when to search and can retry - no explicit evaluation nodes needed.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
• LangChain Retrieval Documentation
• LlamaIndex RAG Documentation
• Pinecone RAG Guide

Further Reading

• OpenAI Embeddings Guide
• Cohere Rerank Documentation
• RAGAS Evaluation Framework
• Meta AI RAG Paper Repository

Summary

• Agentic RAG gives an agent control over retrieval - when to search, what to query, how to evaluate, and when to retry. - It solves multi-hop queries, query-document mismatch, and insufficient context - failures vanilla RAG can't recover from. - Self-evaluation (context relevance, answer faithfulness) is the defining capability - not just multi-pass search.

• Always cap retrieval retries (2–3 max) and route simple queries to vanilla RAG to control cost. - Fix underlying retrieval quality (chunking, hybrid search, reranking) before adding agentic complexity. - Hybrid search + reranking are infrastructure prerequisites, not replaced by agentic patterns. - Evaluate with standard RAG metrics plus agentic-specific metrics: passes per query, reformulation success, routing accuracy.

• Start with create_react_agent + search tool for prototyping; add explicit evaluation nodes for production quality gates.