RAG Guide

TL;DR

• RAG retrieves relevant documents and feeds them to an LLM so it can answer questions grounded in real data instead of relying on training knowledge alone.

• It solves hallucination and knowledge cutoff by injecting fresh, private, or domain-specific context at query time - no retraining required.

• A production RAG pipeline has two phases: offline indexing (load → chunk → embed → store) and online querying (embed query → retrieve → rerank → generate).

• Retrieval quality is the ceiling - most production failures trace back to bad chunking, poor recall, or missing metadata filters, not the LLM.

• Start simple, measure everything - a naive pipeline with solid evaluation beats a complex one without metrics.

Why This Matters

Every AI application that needs to answer questions about specific data - company documents, product catalogs, support tickets, codebases, legal contracts - needs RAG or something like it.

Without RAG, you have two bad options: fine-tune a model on your data (expensive, slow, and the model still hallucinates) or stuff everything into the prompt (impossible once you exceed the context window, and costly at scale).

RAG gives you a third path: keep your data in a searchable index, retrieve only what's relevant, and let the LLM synthesize an answer. This is how most production AI applications work today - from enterprise search to customer support bots to coding assistants.

If you're building anything with LLMs beyond a simple chatbot, you'll encounter RAG. Understanding its architecture, tradeoffs, and failure modes is essential for any AI engineer.

The Problem RAG Solves

Large language models have two fundamental limitations that block production use for domain-specific applications:

Hallucination. LLMs generate fluent text that sounds correct but isn't grounded in fact. Ask GPT-4 about your company's refund policy and it will invent one unless you give it the actual policy text.

Knowledge cutoff and privacy. Models don't know about events after training, proprietary data, or internal systems. You cannot send customer PII to OpenAI for training, and you cannot retrain every time a document changes.

RAG addresses both by separating knowledge storage from reasoning. Your documents live in a searchable index you control. At query time, the system finds relevant passages and passes them to the LLM as context. The model synthesizes an answer from evidence, not from memory.

This pattern also solves operational problems: updating knowledge means re-indexing documents, not retraining models. Swapping GPT-4 for Claude requires changing one API call, not rebuilding a fine-tuned model. Auditors can trace every answer back to source documents.

What Is RAG?

Retrieval-Augmented Generation (RAG) is a pattern where you augment an LLM's input with relevant information retrieved from an external knowledge base. Instead of asking the model to answer from memory, you first search for relevant documents, then pass them alongside the question to the LLM.

The term was introduced in a 2020 paper by Lewis et al. from Meta AI, but the idea - search first, then generate - predates the paper. What made it practical was the combination of dense retrieval (embedding-based search) with large language models capable of synthesizing coherent answers from retrieved passages.

Here's the core idea in pseudocode:

def rag(question, knowledge_base):
    relevant_docs = knowledge_base.search(question, top_k=5)
    context = "\n\n".join(doc.text for doc in relevant_docs)
    prompt = f"""Answer based on the context below. If the context
does not contain the answer, say so.

Context:
{context}

Question: {question}
Answer:"""
    return llm.generate(prompt)

The sophistication is in how well you execute each step - not in the pattern itself.

How RAG Works

A RAG system operates in two distinct phases with different latency, cost, and failure profiles.

Indexing (Offline)

Documents are processed into a searchable representation:

1. Load - Ingest from PDFs, databases, APIs, wikis, code repos.

2. Chunk - Split into retrievable passages (see Chunking Strategies).

3. Embed - Convert each chunk to a vector using an embedding model.

4. Store - Persist vectors, text, and metadata in a vector database.

Querying (Online)

When a user asks a question:

1. Embed the query - Same embedding model as indexing.

2. Retrieve - Find top-k similar chunks via vector search, optionally with hybrid search and metadata filtering.

3. Rerank - Rescore candidates with a cross-encoder reranker.

4. Generate - Pass retrieved context + question to the LLM.

5. Post-process - Add citations, validate faithfulness, log for evaluation.

RAG couples a dense vector index of external knowledge with a sequence-to-sequence generator. At query time, the retriever selects relevant passages and the generator conditions its answer on that evidence.

Source: Meta AI

Architecture

A production RAG system has five layers:

A production RAG stack separates offline indexing (load, chunk, embed, store) from online querying (embed query, retrieve, rerank, generate).

Source: Survey on RAG

Keep ingestion and query paths separate. Re-indexing should not require redeploying your API. Store raw documents separately from vectors so you can re-embed without re-parsing.

Step-by-Step Flow

Building a Production Pipeline

Step 1: Define your corpus and access model. What documents? Who can see what? Metadata for filtering (tenant ID, department, doc type) must be decided before indexing.

Step 2: Ingest and parse. Extract clean text from PDFs (layout-aware parsers like Unstructured or Docling), HTML, markdown, and databases. Preserve structure - headings, page numbers, section IDs.

Step 3: Chunk with structure awareness. Prepend section titles to chunks. Use recursive splitting with overlap. See Chunking Strategies for domain-specific guidance.

Step 4: Embed and upsert. Batch embed chunks (OpenAI, Cohere, or self-hosted). Store vector + text + metadata. Version your embedding model in metadata.

Step 5: Build the retriever. Start with vector search, k=5. Add hybrid search if users search by SKU, error codes, or names. Add metadata filters for multi-tenant isolation.

Step 6: Add reranking. Retrieve top-20, rerank to top-5. This is the highest-ROI quality improvement for most pipelines.

Step 7: Design the generation prompt. Instruct the model to cite sources, refuse when context is insufficient, and answer only from provided context.

Step 8: Instrument and evaluate. Log query, retrieved chunks, scores, and response. Run retrieval evaluation weekly on a golden set.

Chunking Strategies

Chunking is where most RAG pipelines silently fail. The wrong chunk size means the retriever returns too little context (the LLM can't answer) or too much noise (the LLM gets confused).

Tip

Start with recursive chunking at 512 tokens with 50-token overlap. Include the section heading in each chunk. Measure retrieval recall before optimizing chunk size.

Poor chunk boundaries split critical information across chunks - a table header in one chunk and its rows in another. Overlap (10–15% of chunk size) mitigates boundary loss.

Retrieval Techniques

Retrieval determines the ceiling on answer quality. No prompt engineering compensates for missing documents.

Production systems typically use: hybrid retrieve top-20 → metadata filter → rerank to top-5 → generate.

For relationship-heavy queries ("How does component A connect to service B across three systems?"), consider GraphRAG instead of pure vector retrieval.

Real Production Example

A support team at a SaaS company indexes 12,000 help articles, API docs, and release notes. Users ask questions like "Why am I getting a 429 error on the batch endpoint?"

from dataclasses import dataclass
from typing import Optional
import cohere
from openai import OpenAI

@dataclass
class Chunk:
    id: str
    text: str
    metadata: dict

class ProductionRAG:
    def __init__(self, vector_store, embed_model="text-embedding-3-small"):
        self.store = vector_store
        self.openai = OpenAI()
        self.cohere = cohere.Client()
        self.embed_model = embed_model

    def embed(self, text: str) -> list[float]:
        resp = self.openai.embeddings.create(
            model=self.embed_model, input=text
        )
        return resp.data[0].embedding

    def retrieve(
        self,
        query: str,
        tenant_id: str,
        top_k: int = 20,
    ) -> list[Chunk]:
        query_vector = self.embed(query)
        # Hybrid search with tenant isolation
        candidates = self.store.hybrid_search(
            vector=query_vector,
            query_text=query,
            filter={"tenant_id": tenant_id, "status": "published"},
            top_k=top_k,
            alpha=0.7,  # 70% vector, 30% BM25
        )
        return candidates

    def rerank(self, query: str, chunks: list[Chunk], top_n: int = 5) -> list[Chunk]:
        docs = [c.text for c in chunks]
        results = self.cohere.rerank(
            model="rerank-english-v3.0",
            query=query,
            documents=docs,
            top_n=top_n,
        )
        return [chunks[r.index] for r in results.results]

    def generate(self, query: str, chunks: list[Chunk]) -> dict:
        context_parts = []
        for i, chunk in enumerate(chunks, 1):
            source = chunk.metadata.get("url", chunk.metadata.get("title", "unknown"))
            context_parts.append(f"[{i}] ({source})\n{chunk.text}")
        context = "\n\n".join(context_parts)

response = self.openai.chat.completions.create(
            model="gpt-4o",
            temperature=0,
            messages=[
                {"role": "system", "content": (
                    "Answer using ONLY the provided context. "
                    "Cite sources as [1], [2], etc.

"
                    "If the context doesn't contain the answer, say so."
                )},
                {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {query}"},
            ],
        )
        return {
            "answer": response.choices[0].message.content,
            "sources": [c.metadata for c in chunks],
        }

    def query(self, question: str, tenant_id: str) -> dict:
        candidates = self.retrieve(question, tenant_id)
        reranked = self.rerank(question, candidates)
        return self.generate(question, reranked)

# Usage
rag = ProductionRAG(vector_store=my_pinecone_index)
result = rag.query("429 error on batch endpoint", tenant_id="acme-corp")
print(result["answer"])

The tenant_id filter ensures multi-tenant isolation. Hybrid search catches "429" as an exact match while vector search handles "rate limit exceeded." Reranking pushes the API reference above generic troubleshooting articles.

Design Decisions

⚠ Common Mistakes

1. Optimizing prompts before fixing retrieval. If the correct document isn't in top-k, no prompt saves you. Measure recall@k first.

2. Ignoring chunk boundaries. Splitting mid-sentence or separating tables from headers destroys retrieval. Always preserve structure.

3. Skipping metadata. Without tenant filters, users retrieve other customers' documents. Without doc-type filters, release notes drown out API docs.

4. Retrieving too many chunks. k=20 into the LLM adds noise and cost. Retrieve many, rerank to few.

5. No source citations. Without citations, you cannot debug wrong answers or build user trust.

6. Re-embedding without versioning. Changing embedding models without re-indexing creates a mixed vector space where search quality collapses.

7. Treating RAG as set-and-forget. Documents change. Build incremental re-indexing and monitor eval metrics weekly.

Common Failure Modes

Understanding where RAG fails helps you diagnose production issues systematically.

Log every stage. When a user reports a bad answer, trace: was the right doc retrieved? Was it ranked high enough? Did the LLM use it?

Where It Breaks Down

RAG retrieves facts; it does not reason across complex multi-hop relationships. "Which suppliers of our Tier-1 vendor have had compliance violations in the last year?" requires traversing entity relationships - vector search alone will miss connections. Use GraphRAG for this class of query.

RAG also struggles when the answer requires synthesizing information scattered across dozens of documents with no single chunk containing enough context. Summarization-over-retrieval (map-reduce across chunks) adds latency and cost.

Embeddings are lossy. Negation ("not refundable"), numbers ("$4.2M vs $42M"), and domain jargon are common failure points. Hybrid search and reranking mitigate but don't eliminate these.

Finally, RAG cannot answer questions that require real-time data unless your indexing pipeline keeps pace. A 24-hour indexing lag means users get yesterday's truth.

Running RAG in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Important

Fix retrieval before optimizing generation. The number one production mistake is prompt engineering a broken retriever.

Ecosystem

• Orchestration: LangChain, LlamaIndex, Haystack - pipeline frameworks with loaders, splitters, and retrievers.

• Vector Databases: Pinecone, Weaviate, Qdrant, Milvus, Chroma, pgvector.

• Embedding Models: OpenAI text-embedding-3, Cohere embed-v3, BGE, E5, Nomic - see Embedding Models.

• Rerankers: Cohere Rerank, bge-reranker, Jina Reranker - see Re-ranking.

• Evaluation: RAGAS, DeepEval, LangSmith, Braintrust.

• Managed: AWS Bedrock Knowledge Bases, Azure AI Search, Google Vertex AI Search.

Related Technologies

• Chunking Strategies: How you split documents determines retrieval quality.

• Embedding Models: The model that converts text to searchable vectors.

• Hybrid Search: Combining keyword and semantic search for production recall.

• Re-ranking: Cross-encoders that improve precision after initial retrieval.

• Metadata Filtering: Pre-filtering by tenant, date, and document type.

• Retrieval Evaluation: Measuring whether the right documents are found.

• GraphRAG: Knowledge-graph-enhanced retrieval for relationship-heavy queries.

• Vector Databases: Purpose-built storage for embedding vectors.

Learning Path

Prerequisites: Embeddings · Large Language Models

Next topics: Chunking Strategies · Hybrid Search · Re-ranking · Retrieval Evaluation

Estimated time: 50 min · Difficulty: Intermediate

FAQs

When should I use RAG vs. fine-tuning?

Use RAG when you need access to specific, changing data. Use fine-tuning when you need to change behavior, tone, or output format. Many production systems use both.

How many documents can RAG handle?

No hard limit. Vector databases index millions of documents. The bottleneck is embedding cost and retrieval quality at scale, not storage capacity.

What chunk size should I use?

Start with 512 tokens and 50-token overlap. Shorter chunks (256) for factoid Q&A; longer (1024) when answers need surrounding context. Always measure recall@k.

How do I handle frequently changing documents?

Incremental indexing: re-chunk and re-embed only changed documents via upsert. Most vector DBs support this. For real-time data, consider shorter TTL indexes or hybrid with live keyword search.

Can RAG work with PDFs and images?

Yes. Use layout-aware PDF parsers (Unstructured, Docling). For images, use multimodal embeddings or OCR-extracted text. Store page numbers in metadata for citations.

What's the difference between RAG and long-context models?

Long-context models process more text per request but still can't access private data without it in the prompt. RAG retrieves relevant passages - more cost-efficient and works with unlimited corpus sizes. They complement each other.

How do I evaluate RAG quality?

Three layers: retrieval recall@k, faithfulness (answer matches context), and end-to-end correctness. See Retrieval Evaluation and RAGAS for automation.

What causes RAG to fail most often?

Wrong or missing retrieval (60%+ of failures), then chunk fragmentation, then LLM ignoring context. Log each stage to diagnose.

Do I need a vector database or is PostgreSQL enough?

pgvector works well under ~500K vectors and avoids new infrastructure. Dedicated vector DBs offer better ANN performance, hybrid search, and operational features at scale.

How do I add citations?

Pass source metadata with each chunk. Instruct the LLM to cite as [1], [2]. Map citation numbers back to URLs/titles in the response UI.

Should I use an agent instead of a fixed RAG pipeline?

Use a fixed pipeline when queries are straightforward Q&A. Use agentic RAG when the system needs query reformulation, multi-hop retrieval, or self-evaluation. Start fixed; add agent behavior when eval shows retrieval failures that reformulation would fix.

How often should I re-index?

Full re-index when changing embedding models. Incremental re-index on document change events (webhook, CDC, file watcher). Monitor index freshness as a metric.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
• LangChain Retrieval Documentation
• LlamaIndex RAG Documentation
• Pinecone RAG Guide

Further Reading

• OpenAI Embeddings Guide
• Cohere Rerank Documentation
• RAGAS Evaluation Framework
• Meta AI RAG Paper Repository

Summary

• RAG is the standard pattern for grounding LLM answers in your own data.
• The architecture is simple - index, retrieve, generate - but quality depends on every stage.
• Retrieval quality is the ceiling. Fix chunking and search before optimizing prompts.
• Use hybrid search, metadata filters, and reranking in production.
• Always return source citations and run automated evaluation weekly.
• Start simple, measure everything, iterate based on failure analysis.