Retrieval Evaluation Guide

TL;DR

• Retrieval evaluation measures whether the right documents are found - before the LLM ever generates an answer.

• Recall@k is the most important metric - is the correct document in the top-k results? If not, no LLM can save you.

• Build a golden test set of 50–200 query-document pairs from your actual corpus and expected queries.

• Evaluate retrieval separately from generation - conflating them makes it impossible to diagnose which stage failed.

• Run evals in CI - block deploys that regress retrieval metrics. Catch quality drops before users do.

Why This Matters

Most RAG quality problems are retrieval problems. When a user gets a wrong answer, the failure is usually:

1. The correct document wasn't retrieved (retrieval failure - 60%+ of cases)
2. The correct document was retrieved but ranked too low (ranking failure - 20%)
3. The LLM ignored or misinterpreted the context (generation failure - 20%)

If you only evaluate end-to-end answer quality, you can't distinguish between these. You end up prompt-engineering a generation problem that's actually a retrieval problem - or switching embedding models when chunking is the real issue.

Retrieval evaluation isolates the search step, gives you actionable metrics, and lets you iterate on chunking, embedding models, hybrid search, and reranking with clear feedback. It's the difference between guessing and engineering.

The Problem Retrieval Evaluation Solves

RAG pipelines have many configurable components - chunk size, embedding model, search method, reranker, filters. Each change can help or hurt retrieval quality. Without measurement:

• You don't know if a change improved or degraded retrieval
• You can't compare embedding models objectively on your data
• You can't set quality gates in CI/CD
• You can't diagnose user-reported failures systematically
• You optimize the wrong component (prompts instead of chunking)

Retrieval evaluation provides a quantitative feedback loop for the highest-leverage stage in the pipeline. A 5% recall@5 improvement translates directly to fewer wrong answers in production.

What Is Retrieval Evaluation?

Retrieval evaluation measures how well a search system finds relevant documents for a given query. You provide:

1. A test set of queries with known relevant documents (ground truth)
2. A retrieval system to evaluate
3. Metrics that quantify search quality

# Minimal retrieval eval
test_cases = [
    {"query": "refund policy for premium members", "relevant_doc_ids": ["doc_042"]},
    {"query": "API rate limit configuration", "relevant_doc_ids": ["doc_118", "doc_203"]},
    {"query": "SSO setup with Okta", "relevant_doc_ids": ["doc_087"]},
]

for case in test_cases:
    results = retriever.search(case["query"], top_k=10)
    retrieved_ids = [r.id for r in results]
    hit = any(doc_id in retrieved_ids for doc_id in case["relevant_doc_ids"])
    print(f"Query: {case['query']} → {'HIT' if hit else 'MISS'}")

The output tells you whether retrieval works for each query - independent of the LLM.

How Retrieval Evaluation Works

Core Metrics

Recall@k - The Primary Metric

Recall@k = (number of queries where relevant doc appears in top-k) / (total queries)

If recall@5 = 0.80, the correct document appears in the top 5 results for 80% of test queries.

def recall_at_k(retrieved_ids: list[str], relevant_ids: list[str], k: int) -> float:
    top_k = set(retrieved_ids[:k])
    return 1.0 if top_k & set(relevant_ids) else 0.0

def mean_recall_at_k(all_results: list[tuple], k: int) -> float:
    scores = [recall_at_k(retrieved, relevant, k) for retrieved, relevant in all_results]
    return sum(scores) / len(scores)

Target benchmarks:

• recall@5 < 0.60 - retrieval is broken, fix before deploying
• recall@5 0.60–0.80 - acceptable, optimize chunking and search
• recall@5 0.80–0.90 - good, focus on reranking and generation
• recall@5 > 0.90 - excellent, diminishing returns on retrieval

MRR (Mean Reciprocal Rank)

MRR = average of 1/rank_of_first_relevant_document

If the first relevant document is at rank 1, score = 1.0. At rank 3, score = 0.33. MRR rewards getting the right document to the top.

def reciprocal_rank(retrieved_ids: list[str], relevant_ids: list[str]) -> float:
    for rank, doc_id in enumerate(retrieved_ids, 1):
        if doc_id in relevant_ids:
            return 1.0 / rank
    return 0.0

Important when you pass only top-1 or top-3 to the LLM - rank matters, not just presence.

nDCG@k (Normalized Discounted Cumulative Gain)

Accounts for multiple relevant documents and their rank positions. Relevant documents ranked higher score better. Uses graded relevance (highly relevant = 3, partially relevant = 1, irrelevant = 0).

import math

def dcg_at_k(relevances: list[int], k: int) -> float:
    relevances = relevances[:k]
    return sum(rel / math.log2(i + 2) for i, rel in enumerate(relevances))

def ndcg_at_k(retrieved_ids: list[str], relevance_map: dict[str, int], k: int) -> float:
    relevances = [relevance_map.get(doc_id, 0) for doc_id in retrieved_ids[:k]]
    dcg = dcg_at_k(relevances, k)
    ideal = dcg_at_k(sorted(relevance_map.values(), reverse=True), k)
    return dcg / ideal if ideal > 0 else 0.0

Use nDCG when queries have multiple relevant documents with different relevance grades.

A production RAG stack separates offline indexing (load, chunk, embed, store) from online querying (embed query, retrieve, rerank, generate).

Architecture

A retrieval evaluation pipeline has four components:

The retriever scores document chunks against the query embedding and returns the top-k passages that ground the generator.

Source: Meta AI

Run evals on every change to chunking, embedding model, search config, or reranker. Block deploys that drop recall@5 by more than 2%.

Step-by-Step Flow

Step 1: Build a golden test set. Collect 50–200 real queries and identify the correct source document(s) for each. Sources: user query logs, support tickets, manually written test cases.

Step 2: Define relevance criteria. Binary (relevant/not) for recall@k. Graded (0–3) for nDCG. Document what makes a document "relevant" for each query.

Step 3: Run baseline evaluation. Measure recall@5, MRR, and nDCG@5 on your current retrieval pipeline. This is your benchmark.

Step 4: Iterate and compare. Change one component at a time (chunk size, embedding model, hybrid search, reranker). Re-run eval. Keep changes that improve metrics.

Step 5: Automate in CI. Run eval on every PR that touches retrieval code. Fail if recall@5 drops below threshold.

Step 6: Monitor in production. Sample live queries, log retrieved documents, and periodically audit retrieval quality with human review.

Step 7: Expand the test set. Add failure cases from production. Every user-reported wrong answer becomes a new test case.

Real Production Example

An eval pipeline that runs in CI and blocks regressions:

import json
from dataclasses import dataclass

@dataclass
class EvalCase:
    query: str
    relevant_doc_ids: list[str]
    relevance_grades: dict[str, int] = None  # for nDCG

@dataclass
class EvalResult:
    recall_at_5: float
    recall_at_10: float
    mrr: float
    ndcg_at_5: float
    failures: list[dict]

class RetrievalEvaluator:
    def __init__(self, retriever, test_set: list[EvalCase]):
        self.retriever = retriever
        self.test_set = test_set

    def evaluate(self, k_values: list[int] = [5, 10]) -> EvalResult:
        recalls = {k: [] for k in k_values}
        mrrs = []
        ndcgs = []
        failures = []

        for case in self.test_set:
            results = self.retriever.search(case.query, top_k=max(k_values))
            retrieved_ids = [r.id for r in results]

            for k in k_values:
                hit = any(d in retrieved_ids[:k] for d in case.relevant_doc_ids)
                recalls[k].append(1.0 if hit else 0.0)

            rr = 0.0
            for rank, doc_id in enumerate(retrieved_ids, 1):
                if doc_id in case.relevant_doc_ids:
                    rr = 1.0 / rank
                    break
            mrrs.append(rr)

            if case.relevance_grades:
                ndcg = self._ndcg(retrieved_ids, case.relevance_grades, k=5)
                ndcgs.append(ndcg)

            if not any(d in retrieved_ids[:5] for d in case.relevant_doc_ids):
                failures.append({
                    "query": case.query,
                    "expected": case.relevant_doc_ids,
                    "got": retrieved_ids[:5],
                })

        return EvalResult(
            recall_at_5=sum(recalls[5]) / len(recalls[5]),
            recall_at_10=sum(recalls[10]) / len(recalls[10]),
            mrr=sum(mrrs) / len(mrrs),
            ndcg_at_5=sum(ndcgs) / len(ndcgs) if ndcgs else 0.0,
            failures=failures,
        )

    def _ndcg(self, retrieved_ids, relevance_map, k):
        import math
        rels = [relevance_map.get(d, 0) for d in retrieved_ids[:k]]
        dcg = sum(r / math.log2(i + 2) for i, r in enumerate(rels))
        ideal = sorted(relevance_map.values(), reverse=True)[:k]
        idcg = sum(r / math.log2(i + 2) for i, r in enumerate(ideal))
        return dcg / idcg if idcg > 0 else 0.0

    def assert_quality(self, min_recall_at_5: float = 0.75):
        result = self.evaluate()
        print(f"Recall@5:  {result.recall_at_5:.3f}")
        print(f"Recall@10: {result.recall_at_10:.3f}")
        print(f"MRR:       {result.mrr:.3f}")
        print(f"nDCG@5:    {result.ndcg_at_5:.3f}")
        print(f"Failures:  {len(result.failures)}/{len(self.test_set)}")

        if result.failures:
            print("\nFailed queries:")
            for f in result.failures[:5]:
                print(f"  Q: {f['query']}")
                print(f"  Expected: {f['expected']}")
                print(f"  Got: {f['got']}")

        assert result.recall_at_5 >= min_recall_at_5, (
            f"Recall@5 {result.recall_at_5:.3f} below threshold {min_recall_at_5}"
        )

# CI usage
test_set = [EvalCase(**c) for c in json.load(open("golden_test_set.json"))]
evaluator = RetrievalEvaluator(retriever, test_set)
evaluator.assert_quality(min_recall_at_5=0.75)

Design Decisions

⚠ Common Mistakes

1. Only evaluating end-to-end answers. Conflates retrieval and generation failures. Evaluate retrieval independently first.

2. Test set too small. Five queries isn't evaluation - it's anecdote. Minimum 50 cases for statistical confidence.

3. Test set not representative. Only testing easy queries inflates metrics. Include hard cases: paraphrases, ambiguous queries, multi-document answers.

4. Not updating the test set. A static test set goes stale as the corpus changes. Add production failures as new test cases continuously.

5. Changing multiple variables at once. Switching embedding model AND chunk size AND adding hybrid search - you can't attribute the improvement. Change one thing at a time.

6. Ignoring failure analysis. A recall@5 of 0.82 tells you 18% fail. Reading the actual failed queries tells you WHY - and what to fix.

7. Not running evals in CI. Manual evals get skipped under deadline pressure. Automate or it won't happen.

Where It Breaks Down

Subjective relevance - "What is our culture?" may have many partially relevant documents. Binary relevance labels don't capture this well. Use graded relevance or LLM-as-judge with human calibration.

Evolving corpora - Documents are added, updated, and removed. A test case referencing a deleted document becomes invalid. Version your test set alongside your corpus.

Expensive ground truth - Labeling 200 query-document pairs requires domain expertise. Start with 50 high-confidence cases. Expand incrementally from production failure logs.

Non-deterministic retrieval - Approximate nearest neighbor search can return slightly different results across runs. Run evals multiple times or use exact search for evaluation consistency.

Retrieval eval doesn't measure generation. High recall@5 with a bad LLM still produces bad answers. Evaluate both stages - see RAG Evaluation for end-to-end metrics.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Important

Evaluate retrieval before generation. If recall@5 is below 0.70, fix retrieval first - no amount of prompt engineering or LLM upgrading will compensate.

Ecosystem

• RAGAS: Automated RAG evaluation including retrieval metrics (context precision, context recall).

• DeepEval: LLM evaluation framework with retrieval-specific metrics.

• LangSmith: Tracing and evaluation for LangChain pipelines. Dataset management and comparison.

• Braintrust: Eval platform with dataset versioning and regression detection.

• ranx: Python library for ranking evaluation metrics (nDCG, MRR, MAP).

• Pyserini: BM25 evaluation toolkit from Lucene.

Related Technologies

• RAG: The pipeline being evaluated.

• RAG Evaluation: End-to-end evaluation including generation quality.

• Chunking Strategies: First thing to iterate when recall is low.

• Embedding Models: Compare models using retrieval eval.

• Re-ranking: Evaluate with precision@k and nDCG after adding reranker.

• Hybrid Search: A/B test with and without hybrid using recall@k.

Learning Path

Prerequisites: RAG · Embedding Models

Next topics: RAG Evaluation · Re-ranking · Chunking Strategies

Estimated time: 45 min · Difficulty: Intermediate

FAQs

What is the most important retrieval metric?

Recall@k - specifically recall@5 for most RAG pipelines. It answers: "Is the correct document in the top 5 results?" If not, the LLM can't produce a correct answer.

How many test cases do I need?

Minimum 50 for basic confidence. 100–200 for production CI gates. Expand continuously from production failure logs.

How do I build a golden test set?

Collect real user queries (or write representative ones). For each query, identify the correct source document(s) with a domain expert. Store as query → relevant_doc_ids pairs.

Should I evaluate retrieval and generation separately?

Yes. Always. Retrieval eval tells you if the right documents were found. Generation eval tells you if the LLM used them correctly. Combined eval hides which stage failed.

What recall@5 score is good enough?

0.80+ for production. Below 0.60 means retrieval is broken. Between 0.60–0.80, iterate on chunking, embeddings, and search before deploying.

How do I evaluate reranking?

Compare precision@3 and nDCG@5 with and without reranking on the same test set. The reranked pipeline should show higher precision at lower k values.

Can I use LLMs to evaluate retrieval?

Yes - RAGAS uses LLM-as-judge for context precision and recall. Useful for scale, but calibrate against human labels on a sample. LLM judges can be inconsistent.

How often should I run retrieval eval?

On every PR that touches retrieval code (CI gate). Weekly as a scheduled job for monitoring. After any corpus update (new documents, re-indexing).

What is a good MRR score?

MRR > 0.70 means the first relevant document is typically in the top 2 results. MRR > 0.85 is excellent. Low MRR with high recall@5 means the right docs are found but ranked too low - add reranking.

How do I handle queries with multiple relevant documents?

Use nDCG@k with graded relevance scores (0–3). A query about "security policies" may have 3 relevant docs with different relevance levels.

Should I include negative test cases?

Yes. Include queries where the answer is NOT in the corpus. Verify the retriever returns low-scoring results, enabling the system to say "I don't know."

How do I evaluate metadata filtering?

Include test cases that verify tenant isolation (cross-tenant queries return zero results) and permission boundaries (employee queries don't return admin docs).

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
• LangChain Retrieval Documentation
• LlamaIndex RAG Documentation
• Pinecone RAG Guide

Further Reading

• OpenAI Embeddings Guide
• Cohere Rerank Documentation
• RAGAS Evaluation Framework
• Meta AI RAG Paper Repository

Summary

• Retrieval evaluation measures whether the right documents are found - independent of LLM quality.
• Recall@5 is the primary metric. Target 0.80+ for production.
• Build a golden test set of 50–200 real query-document pairs from your corpus.
• Change one retrieval component at a time and re-evaluate.
• Run evals in CI to block deploys that regress retrieval quality.
• Analyze failures individually - aggregate metrics tell you how much; failure cases tell you why.