RAG Evaluation Guide

TL;DR

• RAG evaluation measures retrieval and generation — context precision/recall for search quality; faithfulness and answer relevance for generation; correctness for end-to-end labels when you have them.

• Faithfulness is the critical generation metric — does every claim stick to retrieved context? Unfaithful answers are hallucinations even when retrieval is perfect.

• Evaluate stages independently first — retrieval evaluation isolates search failures; generation metrics isolate LLM/prompt failures. Combined scores hide root cause.

• RAGAS-style pipelines automate the core four — context precision, context recall, faithfulness, answer relevance. Calibrate judges on your domain before CI trust.

• When a metric drops, attribute before you "fix" — low recall → chunking/embeddings/hybrid search/re-ranking; low faithfulness with good recall → prompt/model/hallucination detection; do not swap models for a recall bug.

On this page

• Why This Matters
• The Problem RAG Evaluation Solves
• How We Got Here
• What Is RAG Evaluation?
• How RAG Evaluation Works
• Architecture
• Step-by-Step Flow
• Real Production Example
• Design Decisions
• Comparisons
• Common Mistakes
• Where It Breaks Down
• When NOT to Trust Aggregate RAG Scores
• Running in Production
• Related Guides
• Interview Questions
• Key Takeaways
• FAQs
• References

Why This Matters

Your RAG chatbot answers a refund-policy question with confident, detailed instructions that do not exist in your docs. Retrieval found the right page. The LLM invented the rest.

That is a generation failure, not a retrieval failure. If you only track recall@5, metrics stay green while users get wrong answers. RAG has two failure surfaces; you need metrics for both.

RAG evaluation decomposes the pipeline:

1. Did we retrieve the right context? (context recall, context precision)
2. Did the LLM use that context faithfully? (faithfulness)
3. Did the answer address the question? (answer relevance)
4. Was the end-to-end answer correct? (answer correctness — needs ground truth)

Without stage-specific metrics, teams swap embedding models when the bug is the prompt, or rewrite prompts when chunking strategies broke recall. Evaluation is a diagnostic, not a vanity score.

The Problem RAG Evaluation Solves

A RAG stack combines indexing, retrieval, optional reranking, prompt construction, and generation. Each stage has independent knobs. A change to any one moves end-to-end quality in ways you cannot attribute without structured eval.

Without RAG-specific evaluation:

• You cannot tell retrieval failures from generation failures
• Faithfulness regressions hide until user complaints
• Model or prompt changes break grounding silently
• You cannot compare RAG configs objectively
• CI has no grounded-generation quality gate

RAG evaluation provides stage-aware metrics that map to pipeline components — unit tests vs integration tests for grounded generation. Pair with general LLM evaluation for non-RAG surfaces.

How We Got Here

Early RAG demos measured only "does the answer look right?" Production forced stage metrics once teams discovered lucky hallucinations and silent retrieval misses. The industry converged on the same operational insight as general LLM evaluation: measure the system you ship, then slice failures by stage so ownership is unambiguous.

Diagram: Evolution of RAG measurement

timeline
    title From demo answers to stage-aware RAG eval
    2020 : RAG paper era
         : End-task accuracy on open QA
    2023 : Production RAG boom
         : Recall@k and manual spot checks
    2023-2024 : RAGAS triad / quartet
         : Faithfulness + context metrics
    2024-2026 : CI attribution
         : Failure-stage gates + synthetic sets

End-task scores came first; faithfulness and context metrics followed once fluent wrong answers shipped.

What Is RAG Evaluation?

RAG evaluation measures a retrieval-augmented pipeline across dimensions popularized by RAGAS and related frameworks:

# Conceptual RAG eval case
{
    "question": "What is the refund window for premium members?",
    "contexts": ["Premium members receive full refunds within 60 days..."],
    "answer": "Premium members can request a full refund within 60 days of purchase.",
    "ground_truth": "60 days for premium members",
    "relevant_doc_ids": ["policy_refund_premium"]
}

Scores are typically 0.0–1.0. Aggregate across a golden set; always keep per-stage slices for debugging.

Engineering Insight

Answer correctness alone is insufficient. A parametric lucky guess can pass correctness while failing faithfulness — and will fail on the next question where the corpus differs from training data.

How RAG Evaluation Works

Context recall

Measures whether retrieved chunks contain information needed for the ground-truth answer:

Context Recall = (claims in ground_truth supported by retrieved context) / (total claims in ground_truth)

Low context recall (< 0.70) means retrieval is missing documents. Fix chunking, embeddings, hybrid search, or filters before touching prompts. Implementation: extract claims from ground truth; check entailment against retrieved contexts (NLI or LLM judge).

Context precision

Measures whether retrieved chunks are relevant — penalizes noise:

Context Precision = (relevant chunks in top-k) / k

Ranked variants prefer relevant chunks earlier. High recall with low precision buries the right doc in noise; the generator attends to the wrong spans. Fix with re-ranking or tighter top-k / metadata filters.

Faithfulness

The primary generation metric for grounded systems:

Faithfulness = (answer claims entailed by context) / (total answer claims)

def check_faithfulness(answer: str, contexts: list[str], judge_fn) -> float:
    claims = decompose_claims(answer)
    context_text = "\n".join(contexts)
    supported = sum(1 for c in claims if judge_fn(c, context_text))
    return supported / len(claims) if claims else 1.0

Faithfulness < 0.85 in production is a red flag for factual paths. Users are reading claims not in your corpus — see hallucination detection.

Answer relevance

Does the answer address the question? Catches evasive, off-topic, or incomplete responses that may still be "faithful" (e.g., refusing when evidence exists, or answering a related but different question).

Answer correctness

Compares generated answer to ground truth (semantic similarity, LLM judge, or exact match). Use when labels exist; do not force correctness when the task is multi-valid.

Synthetic test sets

When labeled Q&A is scarce:

1. Sample corpus chunks (stratify by doc type, length, and metadata filters)
2. Generate questions an answerable reader would ask (LLM-assisted — Claude Sonnet 5 or GPT-5.6 Sol work well offline)
3. Keep only questions where a held-out check confirms the chunk supports the answer
4. Inject hard negatives: near-miss docs, outdated versions, and "not in corpus" prompts that should abstain
5. Human-spot-check a stratified slice before CI use

Synthetic sets bootstrap coverage; production logs remain the best long-term source. Version synthetic generators like code. Discard questions that leak answer text into the query (circular eval). Re-generate after major corpus reshapes — a synthetic set tied to last quarter's PDF dump will silently go stale.

Online vs offline RAG evaluation

Offline golden sets are the deploy gate. Online evaluation samples live traffic: log retrieved IDs, faithfulness on a sample, user corrections, and escalation rate. Use online signals to expand the golden set, not to replace it. A faithfulness drop on live traffic without an offline repro usually means distribution shift — new query types your set never covered.

Diagram: Metric drop → what to fix

flowchart TD
    Drop[Metric regression detected] --> Which{Which metric dropped?}
    Which -->|Context recall| Ret[Fix retrieval: chunk embed hybrid rerank]
    Which -->|Context precision| Prec[Tighten top-k / rerank / filters]
    Which -->|Faithfulness| Gen[Fix prompt abstention model verify]
    Which -->|Answer relevance| Rel[Clarify instructions / completeness]
    Which -->|Correctness only| Both[Check faithfulness AND recall first]
    Ret --> ReEval[Re-run stage + e2e eval]
    Prec --> ReEval
    Gen --> ReEval
    Rel --> ReEval
    Both --> ReEval

Attribute before you change: retrieval bugs do not yield to prompt polish.

Architecture

A RAG eval pipeline mirrors the RAG pipeline:

Diagram: RAG evaluation architecture

flowchart TB
    subgraph Corpus [Corpus version]
        Idx[Index + chunking]
    end
    subgraph Online [Online path under test]
        Q[Question] --> Ret[Retrieve + rerank]
        Ret --> Gen[Generate]
    end
    subgraph Eval [Eval harness]
        GT[Golden / synthetic set]
        RS[Retrieval scorers]
        GS[Generation scorers]
        Attr[Failure attribution]
        Gate[CI thresholds]
    end
    Idx --> Ret
    GT --> Q
    Ret --> RS
    Gen --> GS
    RS --> Attr
    GS --> Attr
    Attr --> Gate

Run the full online path in deploy gates so retrieval regressions are visible; use frozen contexts only when isolating generation.

Source: Meta AI

Step-by-Step Flow

Diagram: Diagnosing a single RAG failure

sequenceDiagram
    participant E as Eval runner
    participant R as Retriever
    participant G as Generator
    participant J as Judge / NLI
    E->>R: Question from golden set
    R-->>E: Contexts + IDs
    E->>J: Context recall / precision
    E->>G: Prompt + contexts
    G-->>E: Answer
    E->>J: Faithfulness + relevance + correctness
    E->>E: Tag failure_stage
    Note over E: retrieval | faithfulness | relevance | e2e

Stage tags turn a red dashboard into a backlog owner.

1. Build a golden set — 50–200 questions with ground truth and relevant doc IDs; add "not in corpus" refusal cases.
2. Retrieval baseline — target recall@5 / context recall ≥ 0.80 before heavy generation tuning (retrieval evaluation).
3. Faithfulness scoring — claim decompose + entailment; primary generation metric.
4. Relevance and correctness — catch off-topic and wrong-but-grounded answers.
5. Full baseline — document all metrics for current pipeline + corpus version.
6. CI automation — block if faithfulness < 0.85, context recall < 0.75, or correctness drops > 3%.
7. Diagnose by stage — low recall → retrieval; low faithfulness with good recall → prompt/model; low relevance → instructions.

Real Production Example

End-to-end RAG eval with failure-stage attribution:

from __future__ import annotations

import json
from dataclasses import dataclass
from typing import Any

from openai import OpenAI

client = OpenAI()


@dataclass
class RAGEvalCase:
    question: str
    ground_truth: str
    relevant_doc_ids: list[str]


@dataclass
class RAGEvalResult:
    question: str
    context_recall: float
    faithfulness: float
    answer_relevance: float
    answer_correctness: float
    retrieved_ids: list[str]
    answer: str
    failure_stage: str | None = None


class RAGEvaluator:
    def __init__(self, rag_pipeline, judge_model: str = "gpt-5.6-luna"):
        self.pipeline = rag_pipeline
        self.judge_model = judge_model

    def _judge(self, prompt: str) -> float:
        resp = client.chat.completions.create(
            model=self.judge_model,
            messages=[{"role": "user", "content": prompt}],
            temperature=0,
        )
        try:
            return float(resp.choices[0].message.content.strip())
        except ValueError:
            return 0.0

    def score_context_recall(self, contexts: list[str], ground_truth: str) -> float:
        context_text = "\n".join(contexts)[:4000]
        return self._judge(
            "Score 0.0-1.0: Does the context contain ALL information needed "
            "to produce this ground truth? Return only a float.\n\n"
            f"Context: {context_text}\nGround truth: {ground_truth}"
        )

    def score_faithfulness(self, answer: str, contexts: list[str]) -> float:
        context_text = "\n".join(contexts)[:4000]
        return self._judge(
            "Score 0.0-1.0: Is EVERY claim in the answer supported by the context? "
            "Return only a float.\n\n"
            f"Context: {context_text}\nAnswer: {answer}"
        )

    def score_answer_relevance(self, question: str, answer: str) -> float:
        return self._judge(
            "Score 0.0-1.0: Does the answer fully and directly address the question? "
            "Return only a float.\n\n"
            f"Question: {question}\nAnswer: {answer}"
        )

    def score_correctness(self, answer: str, ground_truth: str) -> float:
        return self._judge(
            "Score 0.0-1.0: Is the answer factually equivalent to the ground truth? "
            "Return only a float.\n\n"
            f"Answer: {answer}\nGround truth: {ground_truth}"
        )

    def evaluate_case(self, case: RAGEvalCase) -> RAGEvalResult:
        result: dict[str, Any] = self.pipeline.query(case.question)
        contexts = result["contexts"]
        answer = result["answer"]
        retrieved_ids = result["retrieved_ids"]

        cr = self.score_context_recall(contexts, case.ground_truth)
        faith = self.score_faithfulness(answer, contexts)
        rel = self.score_answer_relevance(case.question, answer)
        corr = self.score_correctness(answer, case.ground_truth)

        failure_stage = None
        if cr < 0.70:
            failure_stage = "retrieval"
        elif faith < 0.85:
            failure_stage = "generation_faithfulness"
        elif rel < 0.80:
            failure_stage = "generation_relevance"
        elif corr < 0.80:
            failure_stage = "end_to_end"

        return RAGEvalResult(
            question=case.question,
            context_recall=cr,
            faithfulness=faith,
            answer_relevance=rel,
            answer_correctness=corr,
            retrieved_ids=retrieved_ids,
            answer=answer,
            failure_stage=failure_stage,
        )

    def run_eval(self, test_set: list[RAGEvalCase]) -> dict:
        results = [self.evaluate_case(c) for c in test_set]
        metrics = {
            "context_recall": sum(r.context_recall for r in results) / len(results),
            "faithfulness": sum(r.faithfulness for r in results) / len(results),
            "answer_relevance": sum(r.answer_relevance for r in results) / len(results),
            "answer_correctness": sum(r.answer_correctness for r in results) / len(results),
        }
        by_stage: dict[str, int] = {}
        for r in results:
            if r.failure_stage:
                by_stage[r.failure_stage] = by_stage.get(r.failure_stage, 0) + 1

        assert metrics["faithfulness"] >= 0.85
        assert metrics["context_recall"] >= 0.75
        return {"metrics": metrics, "results": results, "failure_breakdown": by_stage}


test_set = [RAGEvalCase(**c) for c in json.load(open("rag_golden_set.json"))]
RAGEvaluator(production_rag_pipeline).run_eval(test_set)

Judge tier note: use a cheap CI judge (GPT-5.6 Luna / Haiku 4.5 / Gemini 3.7 Flash-class) after calibrating against humans or a stronger offline judge (Claude Sonnet 5 / GPT-5.6 Sol). Do not hard-code a permanent "GPT-4o default."

Design Decisions

Common patterns

Decision matrix

When metric drop → fix retrieval vs prompt vs model

Comparisons

Common Mistakes

1. Only measuring end-to-end correctness — lucky hallucinations pass.
2. Skipping retrieval eval — LLM compensates until it does not.
3. Eval with only pre-retrieved context in deploy CI — hides retrieval regressions.
4. Ignoring context precision — noisy context causes synthesis errors.
5. No failure-stage breakdown — 0.82 faithfulness without owners.
6. Static ground truth after corpus updates — version labels with the index.
7. Trusting RAGAS without calibration — correlate with humans on 30+ cases.
8. Swapping models for a recall bug — measure first, then change the right layer.

Common Mistake

Declaring "RAG is broken" and rewriting the prompt when context recall is 0.55. The model cannot faithfully answer from missing evidence.

Where It Breaks Down

• No single ground truth — use faithfulness + relevance; skip forced correctness.
• Multi-hop questions — need explicit multi-doc cases and graded relevance.
• Citation-heavy answers — verify spans, not just prose (hallucination detection).
• Long contexts — judges truncate; prefer claim-level NLI or chunked checks.
• Judge cost — 200 cases with a frontier judge is expensive; cheap CI + weekly deep run.
• Domain correctness — generic faithfulness misses dosage/legal citation errors; add validators.
• Agentic RAG — multi-step retrieval needs per-hop metrics, not only final answer scores.

When NOT to Trust Aggregate RAG Scores

Do not ship or declare victory on a single end-to-end number when:

• Stage metrics disagree — high correctness with low faithfulness is a trap
• Corpus changed without re-labeling the golden set
• Judges are uncalibrated on your domain
• Refusal behavior is untested — missing "not in docs" cases
• You only evaluated frozen contexts while retrieval code changed

Warning

An aggregate "RAG score" without stage attribution will send the wrong team into a week-long rabbit hole.

Running in Production

Best Practice

Gate deploys on faithfulness and context recall separately. Fix retrieval before generation when recall is low.

Production checklist

• Golden set with relevant doc IDs + refusal cases
• Retrieval metrics (recall@k / context recall & precision)
• Faithfulness + answer relevance in CI
• Failure-stage reporting
• Attribution playbook: recall vs faith vs relevance
• Corpus version stamped on every eval run
• Calibrated judge; cheap model in CI (workload-based: Luna / Haiku 4.5 / Gemini 3.7 Flash-class)
• Links to chunking, hybrid search, re-ranking owners
• Re-index / embedding upgrade triggers a mandatory full RAG eval
• Sampled online faithfulness + correction logging feeds new golden cases monthly

Worked attribution examples

These patterns keep teams from treating every red chart as a model problem.

Related Guides

Diagram: RAG evaluation learning path

flowchart LR
    RAG[RAG] --> RetE[Retrieval eval]
    RetE --> RagE[RAG eval]
    RagE --> LE[LLM eval]
    RagE --> HD[Halluc detect]
    RagE --> Chunk[Chunking]
    RagE --> Hyb[Hybrid search]
    RagE --> RR[Re-ranking]

Retrieval eval first; RAG eval binds retrieval to faithfulness; generation fixes follow.

Core:

• RAG — pipeline under test
• Retrieval Evaluation — recall@k, MRR, nDCG
• LLM Evaluation — judges, golden sets, CI patterns
• Hallucination Detection — claim verification
• Evaluation — hub

Levers when metrics drop:

• Chunking Strategies
• Hybrid Search
• Re-ranking

Tools: LangChain · LlamaIndex · ChatGPT · Claude

Interview Questions

1. What metrics matter most for RAG?
   Faithfulness first for grounding; context recall for retrieval adequacy; then relevance and correctness.

2. How is RAG eval different from LLM eval?
   Adds retrieval metrics and faithfulness-to-context; general LLM eval may ignore grounding.

3. Context recall vs precision?
   Recall: was needed info retrieved? Precision: was retrieved info relevant / low-noise?

4. Faithfulness vs correctness?
   Faithfulness: supported by context. Correctness: matches ground truth. Lucky guesses can pass correctness and fail faithfulness.

5. How do you attribute failures?
   Threshold stage metrics; tag retrieval vs faithfulness vs relevance vs e2e; fix that layer.

6. When is synthetic data OK?
   Bootstrap when labels scarce; human-spot-check; prefer production queries long-term.

7. Good faithfulness / recall targets?
   Often faith ≥ 0.85–0.90 and context recall ≥ 0.75–0.80 — tune to risk.

8. Fix prompt or retrieval first?
   If context recall is low, retrieval first. Prompts cannot invent missing evidence reliably.

Key Takeaways

• Score retrieval and generation separately; attribute before changing.
• Faithfulness catches grounded-looking hallucinations; context recall catches missing evidence.
• Use RAGAS-style metrics as a start; calibrate judges on your domain.
• Synthetic sets bootstrap; production failures harden the suite.
• CI should gate stage metrics, not one vanity score.

FAQs

What metrics matter most for RAG?

Faithfulness, then context recall, then answer relevance and correctness (when labeled).

What is a good faithfulness score?

Often ≥ 0.90 for production factual systems; ≥ 0.85 during development. Below 0.80 is systematic hallucination risk.

RAGAS or custom?

Start with RAGAS/DeepEval; custom when you need domain rubrics or citation span checks.

No ground truth?

Use faithfulness + relevance; skip forced correctness.

Target context recall?

Often ≥ 0.80 production; below 0.70 fix retrieval before generation.

How do I diagnose the stage?

Compare recall vs faithfulness vs relevance; use failure_stage tags as in the example.

Eval without an LLM judge?

Retrieval metrics need no judge; faithfulness can use NLI. Judges are more flexible for open answers.

How many cases?

50 minimum; 100–200 for CI; include not-in-corpus refusals.

Multi-document answers?

Graded retrieval relevance; claim-check against the union of contexts.

How often to run?

Every PR touching retrieval/prompts/models; after re-index; weekly drift runs.

Cost in CI?

Roughly a few dollars per 200-case run with a small judge — budget accordingly.

References

• RAGAS Documentation
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
• DeepEval RAG Metrics
• LangChain Retrieval
• LlamaIndex RAG

Further Reading

• Cohere Rerank
• TruLens RAG Triad
• Pinecone RAG Guide