RAG Guide

TL;DR

• RAG retrieves relevant documents and feeds them to an LLM so it can answer questions grounded in real data instead of relying on training knowledge alone.

• It solves hallucination and knowledge cutoff by injecting fresh, private, or domain-specific context at query time - no retraining required.

• A production RAG pipeline has two phases: offline indexing (load → chunk → embed → store) and online querying (embed query → retrieve → rerank → generate).

• Retrieval quality is the ceiling - most operational failures trace back to bad chunking, poor recall, or missing metadata filters, not the LLM.

• Start simple, measure everything - a naive pipeline with solid evaluation beats a complex one without metrics.

Why This Matters

Every AI application that needs to answer questions about specific data - company documents, product catalogs, support tickets, codebases, legal contracts - needs RAG or something like it.

Without RAG, you have two bad options: fine-tune a model on your data (expensive, slow, and the model still hallucinates) or stuff everything into the prompt (impossible once you exceed the context window, and costly at scale).

RAG gives you a third path: keep your data in a searchable index, retrieve only what's relevant, and let the LLM synthesize an answer. This is how most deployed AI applications work today - from enterprise search to customer support bots to coding assistants.

If you're building anything with LLMs beyond a simple chatbot, you'll encounter RAG. Understanding its architecture, tradeoffs, and failure modes is essential for any AI engineer.

The Problem RAG Solves

Large language models have two fundamental limitations that block production use for domain-specific applications:

Hallucination. LLMs generate fluent text that sounds correct but isn't grounded in fact. Ask GPT-4 about your company's refund policy and it will invent one unless you give it the actual policy text.

Knowledge cutoff and privacy. Models don't know about events after training, proprietary data, or internal systems. You cannot send customer PII to OpenAI for training, and you cannot retrain every time a document changes.

RAG addresses both by separating knowledge storage from reasoning. Your documents live in a searchable index you control. At query time, the system finds relevant passages and passes them to the LLM as context. The model synthesizes an answer from evidence, not from memory.

This pattern also solves operational problems: updating knowledge means re-indexing documents, not retraining models. Swapping GPT-4 for Claude requires changing one API call, not rebuilding a fine-tuned model. Auditors can trace every answer back to source documents.

How We Got Here

RAG did not appear fully formed in 2020. It is the convergence of decades of information retrieval research with modern dense embeddings and capable LLMs:

Diagram: Evolution of retrieval-augmented generation

flowchart LR
    A[Keyword IR / BM25] --> B[Dense retrieval / DPR]
    B --> C[RAG paper 2020]
    C --> D[Vector DBs at scale]
    D --> E[Hybrid + rerank pipelines]
    E --> F[Agentic + GraphRAG]

Search moved from lexical matching to dense vectors, then to hybrid fusion, reranking, and graph-augmented retrieval.

Frameworks such as LangChain Retrieval and LlamaIndex RAG standardized the index → retrieve → generate pipeline. The pattern is stable; operational quality still lives in chunking, filtering, and measurement.

What Is RAG?

Retrieval-Augmented Generation (RAG) is a pattern where you augment an LLM's input with relevant information retrieved from an external knowledge base. Instead of asking the model to answer from memory, you first search for relevant documents, then pass them alongside the question to the LLM.

The term was introduced in a 2020 paper by Lewis et al. from Meta AI, but the idea - search first, then generate - predates the paper. What made it practical was the combination of dense retrieval (embedding-based search) with large language models capable of synthesizing coherent answers from retrieved passages.

Here's the core idea in pseudocode:

def rag(question, knowledge_base):
    relevant_docs = knowledge_base.search(question, top_k=5)
    context = "\n\n".join(doc.text for doc in relevant_docs)
    prompt = f"""Answer based on the context below. If the context
does not contain the answer, say so.

Context:
{context}

Question: {question}
Answer:"""
    return llm.generate(prompt)

The sophistication is in how well you execute each step - not in the pattern itself.

How RAG Works

A RAG system operates in two distinct phases with different latency, cost, and failure profiles.

Indexing (Offline)

Documents are processed into a searchable representation:

1. Load - Ingest from PDFs, databases, APIs, wikis, code repos.

2. Chunk - Split into retrievable passages (see Chunking Strategies).

3. Embed - Convert each chunk to a vector using an embedding model.

4. Store - Persist vectors, text, and metadata in a vector database.

Querying (Online)

When a user asks a question:

1. Transform the query (optional) — Rewrite, expand, or generate alternative search queries when the raw question is vague or terminology-mismatched. See Query Transformation.

2. Embed the query - Same embedding model as indexing.

3. Retrieve - Find top-k similar chunks via vector search, optionally with hybrid search and metadata filtering.

4. Rerank - Rescore candidates with a cross-encoder reranker.

5. Generate - Pass retrieved context + question to the LLM.

6. Post-process - Add citations, validate faithfulness, log for evaluation.

Diagram: Online RAG query sequence

sequenceDiagram
    participant U as User
    participant API as RAG API
    participant E as Embedder
    participant V as Vector DB
    participant R as Reranker
    participant L as LLM
    U->>API: question + tenant context
    API->>E: embed query
    E-->>API: query vector
    API->>V: ANN + metadata filter
    V-->>API: top-k chunks
    API->>R: rescore pairs
    R-->>API: top-5 chunks
    API->>L: context + question
    L-->>API: answer + citations
    API-->>U: streamed response

RAG couples a dense vector index of external knowledge with a sequence-to-sequence generator. At query time, the retriever selects relevant passages and the generator conditions its answer on that evidence.

Source: Meta AI

Architecture

A production RAG system has five layers:

Index construction and query execution are usually treated as separate workflows: offline indexing (load, chunk, embed, store) and online querying (embed query, retrieve, rerank, generate).

Source: Survey on RAG

Keep ingestion and query paths separate. Re-indexing should not require redeploying your API. Store raw documents separately from vectors so you can re-embed without re-parsing.

Diagram: Indexing vs query lifecycle

stateDiagram-v2
    [*] --> Ingest: offline
    Ingest --> Chunk
    Chunk --> Embed
    Embed --> Store
    Store --> Ready
    Ready --> Query: online
    Query --> Retrieve
    Retrieve --> Rerank
    Rerank --> Generate
    Generate --> [*]
    Store --> Reindex: model change
    Reindex --> Embed

Step-by-Step Flow

Building a Production Pipeline

Step 1: Define your corpus and access model. What documents? Who can see what? Metadata for filtering (tenant ID, department, doc type) must be decided before indexing.

Step 2: Ingest and parse. Extract clean text from PDFs (layout-aware parsers like Unstructured or Docling), HTML, markdown, and databases. Preserve structure - headings, page numbers, section IDs.

Step 3: Chunk with structure awareness. Prepend section titles to chunks. Use recursive splitting with overlap. See Chunking Strategies for domain-specific guidance.

Step 4: Embed and upsert. Batch embed chunks (OpenAI, Cohere, or self-hosted). Store vector + text + metadata. Version your embedding model in metadata.

Step 5: Build the retriever. Start with vector search, k=5. Add hybrid search if users search by SKU, error codes, or names. Add metadata filters for multi-tenant isolation.

Step 6: Add reranking. Retrieve top-20, rerank to top-5. This is one of the highest-ROI quality improvements for most pipelines.

Step 7: Design the generation prompt. Instruct the model to cite sources, refuse when context is insufficient, and answer only from provided context.

Step 8: Instrument and evaluate. Log query, retrieved chunks, scores, and response. Run retrieval evaluation weekly on a golden set.

Chunking Strategies

Chunking is where most RAG pipelines silently fail. The wrong chunk size means the retriever returns too little context (the LLM can't answer) or too much noise (the LLM gets confused).

Tip

Start with recursive chunking at 512 tokens with 50-token overlap. Include the section heading in each chunk. Measure retrieval recall before optimizing chunk size.

Poor chunk boundaries split critical information across chunks - a table header in one chunk and its rows in another. Overlap (10–15% of chunk size) mitigates boundary loss.

Retrieval Techniques

Retrieval determines the ceiling on answer quality. No prompt engineering compensates for missing documents.

Production systems typically use: hybrid retrieve top-20 → metadata filter → rerank to top-5 → generate.

For relationship-heavy queries ("How does component A connect to service B across three systems?"), consider GraphRAG instead of pure vector retrieval.

Real Production Example

A support team at a SaaS company indexes 12,000 help articles, API docs, and release notes. Users ask questions like "Why am I getting a 429 error on the batch endpoint?"

from dataclasses import dataclass
from typing import Optional
import cohere
from openai import OpenAI

@dataclass
class Chunk:
    id: str
    text: str
    metadata: dict

class ProductionRAG:
    def __init__(self, vector_store, embed_model="text-embedding-3-small"):
        self.store = vector_store
        self.openai = OpenAI()
        self.cohere = cohere.Client()
        self.embed_model = embed_model

    def embed(self, text: str) -> list[float]:
        resp = self.openai.embeddings.create(
            model=self.embed_model, input=text
        )
        return resp.data[0].embedding

    def retrieve(
        self,
        query: str,
        tenant_id: str,
        top_k: int = 20,
    ) -> list[Chunk]:
        query_vector = self.embed(query)
        # Hybrid search with tenant isolation
        candidates = self.store.hybrid_search(
            vector=query_vector,
            query_text=query,
            filter={"tenant_id": tenant_id, "status": "published"},
            top_k=top_k,
            alpha=0.7,  # 70% vector, 30% BM25
        )
        return candidates

    def rerank(self, query: str, chunks: list[Chunk], top_n: int = 5) -> list[Chunk]:
        docs = [c.text for c in chunks]
        results = self.cohere.rerank(
            model="rerank-english-v3.0",
            query=query,
            documents=docs,
            top_n=top_n,
        )
        return [chunks[r.index] for r in results.results]

    def generate(self, query: str, chunks: list[Chunk]) -> dict:
        context_parts = []
        for i, chunk in enumerate(chunks, 1):
            source = chunk.metadata.get("url", chunk.metadata.get("title", "unknown"))
            context_parts.append(f"[{i}] ({source})\n{chunk.text}")
        context = "\n\n".join(context_parts)

response = self.openai.chat.completions.create(
            model="gpt-4o",
            temperature=0,
            messages=[
                {"role": "system", "content": (
                    "Answer using ONLY the provided context. "
                    "Cite sources as [1], [2], etc.

"
                    "If the context doesn't contain the answer, say so."
                )},
                {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"},
            ],
        )
        return {
            "answer": response.choices[0].message.content,
            "sources": [c.metadata for c in chunks],
        }

    def query(self, question: str, tenant_id: str) -> dict:
        candidates = self.retrieve(question, tenant_id)
        reranked = self.rerank(question, candidates)
        return self.generate(question, reranked)

# Usage
rag = ProductionRAG(vector_store=my_pinecone_index)
result = rag.query("429 error on batch endpoint", tenant_id="acme-corp")
print(result["answer"])

The tenant_id filter ensures multi-tenant isolation. Hybrid search catches "429" as an exact match while vector search handles "rate limit exceeded." Reranking pushes the API reference above generic troubleshooting articles.

Design Decisions

Comparisons

RAG vs fine-tuning

Many production systems combine both: RAG for facts, fine-tuning for tone and structured output.

RAG vs long-context prompting

Stuffing entire corpora into a 128K+ context window avoids retrieval infrastructure but scales poorly on cost and latency. RAG retrieves only relevant passages—typically 2–8K tokens of context—making unlimited corpus sizes practical.

RAG vs GraphRAG

RAG vs agents

RAG is a retrieval pattern. An agent may call retrieval as one tool among many. Agentic RAG adds a loop that reformulates queries, selects indexes, or self-evaluates retrieval—use it when fixed pipelines fail evals, not by default.

Decision tree: do you need RAG?

Decision tree: When to use RAG

flowchart TD
    A[Need answers from your data?] -->|No| B[General LLM / chatbot]
    A -->|Yes| C[Data changes or is private?]
    C -->|No| D[Fine-tune or prompt with public corpus]
    C -->|Yes| E[Relationship-heavy queries?]
    E -->|Yes| F[Consider GraphRAG + vector RAG]
    E -->|No| G[Standard RAG pipeline]
    G --> H{Exact IDs / SKUs in queries?}
    H -->|Yes| I[Add hybrid search + filters]
    H -->|No| J[Dense + rerank baseline]
    I --> K[Measure recall@k weekly]
    J --> K

Start with dense retrieval + rerank; add hybrid search, filters, and graph augmentation only when evals show specific gaps.

Head-to-head tooling

Compare stacks in Best Vector Databases: Qdrant vs Pinecone · Pinecone vs Weaviate · Milvus vs Qdrant.

Common Mistakes

1. Optimizing prompts before fixing retrieval. If the correct document isn't in top-k, no prompt saves you. Measure recall@k first.

2. Ignoring chunk boundaries. Splitting mid-sentence or separating tables from headers destroys retrieval. Always preserve structure.

3. Skipping metadata. Without tenant filters, users retrieve other customers' documents. Without doc-type filters, release notes drown out API docs.

4. Retrieving too many chunks. k=20 into the LLM adds noise and cost. Retrieve many, rerank to few.

5. No source citations. Without citations, you cannot debug wrong answers or build user trust.

6. Re-embedding without versioning. Changing embedding models without re-indexing creates a mixed vector space where search quality collapses.

7. Treating RAG as set-and-forget. Documents change. Build incremental re-indexing and monitor eval metrics weekly.

Common Failure Modes

Understanding where RAG fails helps you diagnose production issues systematically.

Log every stage. When a user reports a bad answer, trace: was the right doc retrieved? Was it ranked high enough? Did the LLM use it?

Where It Breaks Down

RAG retrieves facts; it does not reason across complex multi-hop relationships. "Which suppliers of our Tier-1 vendor have had compliance violations in the last year?" requires traversing entity relationships - vector search alone will miss connections. Use GraphRAG for this class of query.

RAG also struggles when the answer requires synthesizing information scattered across dozens of documents with no single chunk containing enough context. Summarization-over-retrieval (map-reduce across chunks) adds latency and cost.

Embeddings are lossy. Negation ("not refundable"), numbers ("$4.2M vs $42M"), and domain jargon are common failure points. Hybrid search and reranking mitigate but don't eliminate these.

Finally, RAG cannot answer questions that require real-time data unless your indexing pipeline keeps pace. A 24-hour indexing lag means users get yesterday's truth.

When NOT to Use RAG

Skip building a RAG pipeline when:

1. Answers live entirely in public, static knowledge — a capable base model with web search or a small curated prompt may suffice.
2. The task is pure generation or transformation — summarization of provided text, translation, or code completion without external corpus lookup.
3. You need guaranteed reasoning over entity graphs — multi-hop "who supplies whom" questions need GraphRAG or structured queries, not vector similarity alone.
4. Sub-100ms latency is mandatory — full RAG (embed + search + rerank + generate) typically exceeds 1s; cache or precompute instead.
5. You cannot enforce access control at retrieval — if metadata filters and tenant isolation are impossible, do not expose mixed corpora through an LLM.
6. Corpus is tiny and static — under ~50 pages, long-context or a single system prompt may be simpler with equivalent quality.

Prefer fine-tuning when the goal is behavior and format, not factual lookup. Prefer agents when retrieval strategy itself must adapt per query.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Important

Fix retrieval before optimizing generation. A common operational mistake is prompt engineering a broken retriever.

Related Guides

• Models: GPT-5 · Claude Sonnet · Claude Haiku · Gemini 3.1 Pro · Llama 4 — common generators on top of retrieved context.

• Companies: OpenAI · Anthropic · Hugging Face — model providers and open embedding ecosystems for RAG stacks.

• Foundations: Embeddings · Embedding Models · Chunking Strategies · Large Language Models

• Retrieval stack: Vector Search · Hybrid Search · Metadata Filtering · ANN Indexes · Vector Quantization · Late Interaction Retrieval · Re-ranking

• Quality & ops: Retrieval Evaluation · Vector Databases

• Advanced patterns: Agentic RAG · GraphRAG

• Orchestration: LangChain · LlamaIndex — pipeline frameworks with loaders, splitters, and retrievers.

• Vector stores: Qdrant · Weaviate · Pinecone · Milvus · pgvector · Chroma — compare in Best Vector Databases.

• Head-to-heads: Qdrant vs Pinecone · Pinecone vs Weaviate · Milvus vs Qdrant · pgvector vs Pinecone

If you understood this topic, read next:

Diagram: Recommended learning path

flowchart LR
    A[Embeddings] --> B[Chunking]
    B --> C[RAG]
    C --> D[Hybrid]
    D --> E[Rerank]
    E --> F[Eval]

Prerequisites: Embeddings · Large Language Models

Next topics: Chunking Strategies · Hybrid Search · Re-ranking · Retrieval Evaluation

Estimated time: 50 min · Difficulty: Intermediate

Interview Questions

1. What problem does RAG solve that fine-tuning does not?

   • Expected: fresh/private data at query time, citeable sources, no full retrain on doc changes; fine-tuning changes behavior not live facts.

2. Where does retrieval quality cap overall RAG quality?

   • Expected: if correct chunk ∉ top-k, generation cannot recover; measure recall@k before prompt tuning (Retrieval Evaluation).

3. When would you add hybrid search over dense-only?

   • Expected: SKUs, error codes, rare tokens, exact policy numbers—BM25 + dense fusion (Hybrid Search).

4. What metadata must you filter at retrieval time for multi-tenant SaaS?

   • Expected: tenant_id, doc ACL, sensitivity; never rely on LLM to self-filter (Metadata Filtering).

5. Why rerank after ANN retrieval?

   • Expected: bi-encoder recall vs cross-encoder precision; retrieve 20–50, rerank to 5 (Re-ranking).

6. What triggers a full re-index vs incremental upsert?

   • Expected: embedding model version change → full re-index; single doc edit → incremental upsert with versioned metadata.

7. RAG vs GraphRAG — when do you escalate?

   • Expected: multi-hop relationship queries across entities; vector RAG for passage similarity Q&A (GraphRAG).

8. Name three production monitoring signals for RAG.

   • Expected: recall@k on golden set, faithfulness/citation rate, p95 latency per stage, filter miss rate, index freshness.

Key Takeaways

• RAG is the standard pattern for grounding LLM answers in your own data.
• The architecture is simple — index, retrieve, generate — but quality depends on every stage.
• Retrieval quality is the ceiling. Fix chunking and search before optimizing prompts.
• Use hybrid search, metadata filters, and reranking in production.
• Always return source citations and run automated evaluation weekly.
• Start simple, measure everything, iterate based on failure analysis.
• Compare vector stores in Best Vector Databases before committing to infrastructure.

FAQs

When should I use RAG vs. fine-tuning?

Use RAG when you need access to specific, changing data. Use fine-tuning when you need to change behavior, tone, or output format. Many production systems use both.

How many documents can RAG handle?

No hard limit. Vector databases index millions of documents. The bottleneck is embedding cost and retrieval quality at scale, not storage capacity.

What chunk size should I use?

Start with 512 tokens and 50-token overlap. Shorter chunks (256) for factoid Q&A; longer (1024) when answers need surrounding context. Always measure recall@k.

How do I handle frequently changing documents?

Incremental indexing: re-chunk and re-embed only changed documents via upsert. Most vector DBs support this. For real-time data, consider shorter TTL indexes or hybrid with live keyword search.

Can RAG work with PDFs and images?

Yes. Use layout-aware PDF parsers (Unstructured, Docling). For images, use multimodal embeddings or OCR-extracted text. Store page numbers in metadata for citations.

What's the difference between RAG and long-context models?

Long-context models process more text per request but still can't access private data without it in the prompt. RAG retrieves relevant passages - more cost-efficient and works with unlimited corpus sizes. They complement each other.

How do I evaluate RAG quality?

Three layers: retrieval recall@k, faithfulness (answer matches context), and end-to-end correctness. See Retrieval Evaluation and RAGAS for automation.

What causes RAG to fail most often?

Wrong or missing retrieval is often the dominant failure mode in RAG pipelines, followed by chunk fragmentation and LLM ignoring context. Log each stage to diagnose.

Do I need a vector database or is PostgreSQL enough?

pgvector with HNSW/IVFFlat is often a good fit when you already run PostgreSQL at moderate scale. Dedicated vector DBs typically offer richer hybrid search, filtering, and operational features as requirements grow.

How do I add citations?

Pass source metadata with each chunk. Instruct the LLM to cite as [1], [2]. Map citation numbers back to URLs/titles in the response UI.

Should I use an agent instead of a fixed RAG pipeline?

Use a fixed pipeline when queries are straightforward Q&A. Use agentic RAG when the system needs query reformulation, multi-hop retrieval, or self-evaluation. Start fixed; add agent behavior when eval shows retrieval failures that reformulation would fix.

How often should I re-index?

Full re-index when changing embedding models. Incremental re-index on document change events (webhook, CDC, file watcher). Monitor index freshness as a metric.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
• LangChain Retrieval Documentation
• LlamaIndex RAG Documentation
• Pinecone RAG Guide

Further Reading

• OpenAI Embeddings Guide
• Cohere Rerank Documentation
• RAGAS Evaluation Framework
• Meta AI RAG Paper Repository