RAG Evaluation Guide

TL;DR

• RAG evaluation measures both retrieval and generation - context precision/recall for retrieval, faithfulness and answer relevance for generation.

• Faithfulness is the critical metric - does the answer stick to retrieved context? Unfaithful answers are hallucinations even when retrieval is perfect.

• Evaluate stages independently first - retrieval eval isolates search failures; generation eval isolates LLM failures.

Combined metrics hide root cause.

• RAGAS automates the core metrics - context precision, context recall, faithfulness, answer relevance. Good starting point; calibrate on your domain.

• Run end-to-end eval in CI - block deploys where faithfulness drops below 0.85 or context recall drops below 0.75.

Why This Matters

Your RAG chatbot answers a question about refund policy with confident, detailed instructions - that don't exist in your documentation. Retrieval found the right policy page. The LLM invented the rest.

This is a generation failure, not a retrieval failure. If you only measure recall@5, you'd report green metrics while users get wrong answers. RAG has two failure surfaces, and you need metrics for both.

RAG evaluation decomposes the pipeline into measurable stages:

1. Did we retrieve the right context? (context recall, context precision)
2. Did the LLM use that context faithfully? (faithfulness)
3. Did the answer actually address the question? (answer relevance)
4. Was the end-to-end answer correct? (answer correctness)

Without stage-specific metrics, you optimize blindly. Teams swap embedding models when the problem is prompt design. They rewrite prompts when chunking is the issue. RAG evaluation gives you a diagnostic - not just a score.

The Problem RAG Evaluation Solves

RAG pipelines combine retrieval, reranking, prompt construction, and generation. Each component has independent configuration knobs. A change to any one affects end-to-end quality in ways that are hard to attribute without structured evaluation.

Without RAG-specific evaluation:

• You cannot distinguish retrieval failures from generation failures
• Faithfulness regressions go undetected until user complaints
• Model or prompt changes break grounding silently
• You cannot compare RAG configurations objectively
• CI has no quality gate specific to grounded generation

RAG evaluation provides stage-aware metrics that map directly to pipeline components - the debugging equivalent of unit tests vs integration tests.

What Is RAG Evaluation?

RAG evaluation measures the quality of a retrieval-augmented generation pipeline across four dimensions defined by the RAGAS framework (and widely adopted):

# Conceptual RAG eval case
{
    "question": "What is the refund window for premium members?",
    "contexts": ["Premium members receive full refunds within 60 days..."],
    "answer": "Premium members can request a full refund within 60 days of purchase.",
    "ground_truth": "60 days for premium members"
}

Each metric scores 0.0–1.0. Aggregate across your golden test set for pipeline-level scores.

How RAG Evaluation Works

Context Recall

Measures whether retrieved chunks contain the information needed to answer the question.

Context Recall = (claims in ground_truth supported by retrieved context) / (total claims in ground_truth)

Low context recall (< 0.70) means retrieval is missing documents. Fix chunking, embeddings, or search before touching prompts.

Implementation: LLM extracts claims from ground truth answer, checks each against retrieved contexts via entailment or LLM judge.

Context Precision

Measures whether retrieved chunks are relevant - penalizes noise in context.

Context Precision = (relevant chunks in top-k) / k

Or ranked version: relevant chunks should appear before irrelevant ones.

High recall with low precision means the right doc is retrieved but buried in noise. The LLM may attend to wrong chunks. Fix with reranking or tighter top-k.

Faithfulness

The most important generation metric. Measures whether every claim in the answer is supported by retrieved context.

Faithfulness = (answer claims entailed by context) / (total answer claims)

def check_faithfulness(answer: str, contexts: list[str], judge_fn) -> float:
    claims = decompose_claims(answer)  # LLM extracts atomic claims
    context_text = "\n".join(contexts)
    supported = sum(1 for c in claims if judge_fn(c, context_text))
    return supported / len(claims) if claims else 1.0

Faithfulness < 0.85 in production is a red flag. Users are getting information not in your corpus.

Answer Relevance

Measures whether the answer addresses the question - catches evasive, off-topic, or incomplete responses.

Answer Relevance = semantic similarity(question, answer) adjusted for completeness

Or LLM judge: "Does this answer fully address the question? Score 0-1."

Low answer relevance with high faithfulness means the model is grounded but unhelpful - possibly refusing to answer or giving partial responses.

Answer Correctness

End-to-end metric comparing generated answer to ground truth. Combines retrieval and generation quality.

Uses semantic similarity, LLM-as-judge, or exact match depending on answer type.

A production RAG stack separates offline indexing (load, chunk, embed, store) from online querying (embed query, retrieve, rerank, generate).

Architecture

A RAG evaluation pipeline mirrors the RAG pipeline itself:

The retriever scores document chunks against the query embedding and returns the top-k passages that ground the generator.

Source: Meta AI

Run retrieval metrics and generation metrics separately when debugging. Run end-to-end when validating deploys.

Step-by-Step Flow

Step 1: Build a golden test set. Collect 50–200 questions with ground truth answers and known relevant document IDs. Source from user logs, support tickets, domain experts.

Step 2: Establish retrieval baseline. Run retrieval evaluation first. Target recall@5 ≥ 0.80 before evaluating generation.

Step 3: Implement faithfulness scoring. Decompose answers into claims. Verify each claim against retrieved context. This is your primary generation metric.

Step 4: Add answer relevance and correctness. Catch off-topic and incorrect answers that are technically faithful.

Step 5: Run full RAG eval baseline. Score all metrics on current pipeline. Document as benchmark.

Step 6: Automate in CI. Block deploys if faithfulness < 0.85, context recall < 0.75, or answer correctness drops > 3%.

Step 7: Diagnose failures by stage. Low context recall → fix retrieval. Low faithfulness with good recall → fix prompts or model. Low relevance → fix prompt instructions.

Real Production Example

End-to-end RAG eval pipeline with stage-specific diagnostics:

import json
from dataclasses import dataclass, field
from openai import OpenAI

client = OpenAI()

@dataclass
class RAGEvalCase:
    question: str
    ground_truth: str
    relevant_doc_ids: list[str]

@dataclass
class RAGEvalResult:
    question: str
    context_recall: float
    faithfulness: float
    answer_relevance: float
    answer_correctness: float
    retrieved_ids: list[str]
    answer: str
    failure_stage: str | None = None

class RAGEvaluator:
    def __init__(self, rag_pipeline, judge_model: str = "gpt-4o-mini"):
        self.pipeline = rag_pipeline
        self.judge_model = judge_model

    def _judge(self, prompt: str) -> float:
        resp = client.chat.completions.create(
            model=self.judge_model,
            messages=[{"role": "user", "content": prompt}],
            temperature=0,
        )
        try:
            return float(resp.choices[0].message.content.strip())
        except ValueError:
            return 0.0

    def score_context_recall(self, contexts: list[str], ground_truth: str) -> float:
        context_text = "\n".join(contexts)[:4000]
        return self._judge(f"""Score 0.0-1.0: Does the context contain ALL information
needed to produce this ground truth answer? Return only a float.

Context: {context_text}
Ground truth: {ground_truth}""")

    def score_faithfulness(self, answer: str, contexts: list[str]) -> float:
        context_text = "\n".join(contexts)[:4000]
        return self._judge(f"""Score 0.0-1.0: Is EVERY claim in the answer
supported by the context? Penalize any unsupported claim heavily. Return only a float.

Context: {context_text}
Answer: {answer}""")

    def score_answer_relevance(self, question: str, answer: str) -> float:
        return self._judge(f"""Score 0.0-1.0: Does the answer fully and directly
address the question? Return only a float.

Question: {question}
Answer: {answer}""")

    def score_correctness(self, answer: str, ground_truth: str) -> float:
        return self._judge(f"""Score 0.0-1.0: Is the answer factually equivalent
to the ground truth? Return only a float.

Answer: {answer}
Ground truth: {ground_truth}""")

    def evaluate_case(self, case: RAGEvalCase) -> RAGEvalResult:
        result = self.pipeline.query(case.question)
        contexts = result["contexts"]
        answer = result["answer"]
        retrieved_ids = result["retrieved_ids"]

        cr = self.score_context_recall(contexts, case.ground_truth)
        faith = self.score_faithfulness(answer, contexts)
        rel = self.score_answer_relevance(case.question, answer)
        corr = self.score_correctness(answer, case.ground_truth)

        failure_stage = None
        if cr < 0.70:
            failure_stage = "retrieval"
        elif faith < 0.85:
            failure_stage = "generation_faithfulness"
        elif rel < 0.80:
            failure_stage = "generation_relevance"
        elif corr < 0.80:
            failure_stage = "end_to_end"

        return RAGEvalResult(
            question=case.question,
            context_recall=cr,
            faithfulness=faith,
            answer_relevance=rel,
            answer_correctness=corr,
            retrieved_ids=retrieved_ids,
            answer=answer,
            failure_stage=failure_stage,
        )

    def run_eval(self, test_set: list[RAGEvalCase]) -> dict:
        results = [self.evaluate_case(c) for c in test_set]

        metrics = {
            "context_recall": sum(r.context_recall for r in results) / len(results),
            "faithfulness": sum(r.faithfulness for r in results) / len(results),
            "answer_relevance": sum(r.answer_relevance for r in results) / len(results),
            "answer_correctness": sum(r.answer_correctness for r in results) / len(results),
        }

        by_stage = {}
        for r in results:
            if r.failure_stage:
                by_stage[r.failure_stage] = by_stage.get(r.failure_stage, 0) + 1

        print("RAG Eval Results:")
        for k, v in metrics.items():
            print(f"  {k}: {v:.3f}")
        print(f"\nFailure breakdown: {by_stage}")

        assert metrics["faithfulness"] >= 0.85, f"Faithfulness {metrics['faithfulness']:.3f} below 0.85"
        assert metrics["context_recall"] >= 0.75, f"Context recall {metrics['context_recall']:.3f} below 0.75"
        return {"metrics": metrics, "results": results, "failure_breakdown": by_stage}

# CI usage
test_set = [RAGEvalCase(**c) for c in json.load(open("rag_golden_set.json"))]
evaluator = RAGEvaluator(production_rag_pipeline)
evaluator.run_eval(test_set)

Design Decisions

⚠ Common Mistakes

1. Only measuring end-to-end correctness. A correct answer from hallucination (lucky guess) passes eval but will fail on other queries. Always measure faithfulness.

2. Skipping retrieval eval. Generation metrics look fine because the LLM compensates for bad retrieval - until it doesn't. Evaluate retrieval first.

3. Evaluating with pre-retrieved context. Hides retrieval regressions. Run the full pipeline in eval, including retrieval.

4. Ignoring context precision. High recall with noisy context causes the LLM to synthesize across irrelevant chunks. Measure precision alongside recall.

5. Not diagnosing failure stage. Aggregate faithfulness of 0.82 tells you there's a problem. Failure stage breakdown tells you it's 70% retrieval, 30% generation.

6. Static ground truth. Ground truth answers become stale as corpus updates. Version ground truth with corpus versions.

7. Trusting RAGAS scores without calibration. Run RAGAS on 30 cases, have humans score the same cases. Measure correlation before using in CI.

Where It Breaks Down

No ground truth available - Many production queries lack a single correct answer. Use faithfulness and answer relevance instead of correctness. LLM-as-judge with rubrics fills the gap.

Multi-hop reasoning - Single-turn RAG eval misses questions requiring synthesis across multiple retrieved chunks. Build multi-hop test cases explicitly.

Citation-heavy answers - Faithfulness scoring must verify citations, not just answer text. See Hallucination Detection.

Long contexts - LLM judges truncate or lose track of long retrieved contexts. Chunk eval or use NLI models for entailment checking.

RAGAS cost - Full RAGAS eval on 200 cases with GPT-4 judge costs $5–20 per run. Use cheaper judge models in CI; run full eval weekly.

Domain-specific correctness - Generic faithfulness doesn't catch domain errors (wrong medical dosage, incorrect legal citation). Add domain-specific validators.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Important

Fix retrieval before generation. Context recall below 0.70 means no prompt engineering will consistently produce faithful answers - the necessary information isn't reaching the LLM.

Ecosystem

• RAGAS: Reference implementation for context precision, context recall, faithfulness, answer relevance. Python library with LangChain integration.

• DeepEval: RAG metrics including contextual precision/recall, faithfulness, answer relevancy. Pytest integration for CI.

• LangSmith: Dataset management and eval runs for LangChain RAG chains. Trace-linked evaluation.

• Braintrust: Scoring functions for RAG metrics.

Dataset versioning and regression comparison across pipeline versions.

• TruLens: RAG triad metrics (context relevance, groundedness, answer relevance) with feedback functions.

• Arize Phoenix: Open-source observability with RAG evaluation metrics and drift detection.

Related Technologies

• RAG: The pipeline being evaluated.

• Retrieval Evaluation: Evaluate retrieval stage independently with recall@k, MRR, nDCG.

• LLM Evaluation: General LLM output evaluation including LLM-as-judge patterns.

• Hallucination Detection: Specialized faithfulness and claim verification.

• Re-ranking: Evaluate context precision improvement after adding reranker.

• Chunking Strategies: First lever when context recall is low.

Learning Path

Prerequisites: RAG · Retrieval Evaluation · LLM Evaluation

Next topics: Hallucination Detection · Re-ranking · Agentic RAG

Estimated time: 50 min · Difficulty: Intermediate

FAQs

What metrics matter most for RAG?

Faithfulness first - is the answer grounded in retrieved context? Then context recall - was the right information retrieved? Then answer relevance and correctness. In that order.

What is a good faithfulness score?

≥ 0.90 for production factual systems. ≥ 0.85 acceptable during development. Below 0.80 means systematic hallucination - do not deploy.

How is RAG evaluation different from LLM evaluation?

LLM evaluation measures output quality generally. RAG evaluation adds retrieval-specific metrics (context recall, context precision) and faithfulness - grounding in retrieved context specifically.

Should I use RAGAS or build custom eval?

Start with RAGAS for standard metrics. Build custom scorers when you need domain-specific rubrics, citation verification, or integration with existing CI infrastructure.

How do I evaluate RAG when I don't have ground truth answers?

Use faithfulness (grounded in context) and answer relevance. Skip answer correctness. LLM-as-judge with rubrics works without reference answers.

What context recall score should I target?

≥ 0.80 for production. Below 0.70 means retrieval is failing - fix chunking, embeddings, or search before evaluating generation.

How do I diagnose which RAG stage failed?

Compare stage metrics: low context recall → retrieval. Low faithfulness with good recall → generation/prompt. Low relevance with good faithfulness → prompt instructions. Failure stage breakdown in eval results makes this automatic.

Can I run RAG eval without calling an LLM judge?

Partially. Retrieval metrics (recall@k) need no LLM. Faithfulness can use NLI entailment models (cheaper, faster, less accurate). LLM judge is most flexible for open-ended answers.

How many RAG test cases do I need?

50 minimum. 100–200 for CI gates. Include cases where the answer is NOT in the corpus (should refuse or say "I don't know").

How do I eval multi-document RAG answers?

Use nDCG-style graded relevance for retrieval. For generation, decompose the answer into claims and verify each against the union of retrieved contexts.

Does RAGAS work with any LLM provider?

RAGAS supports OpenAI, Anthropic, and local models via LangChain integrations. Configure the LLM used for metric computation separately from your RAG pipeline LLM.

How often should I run RAG eval?

Every PR touching retrieval, prompts, or models. After corpus updates (re-indexing). Weekly scheduled run for drift monitoring.

What's the cost of RAG eval in CI?

~$2–10 per run for 200 cases with gpt-4o-mini judge. ~$20–50 with gpt-4o. Run cheap judge in CI; full judge weekly.

References

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
• LangChain Retrieval Documentation
• LlamaIndex RAG Documentation
• Pinecone RAG Guide

Further Reading

• OpenAI Embeddings Guide
• Cohere Rerank Documentation
• RAGAS Evaluation Framework
• Meta AI RAG Paper Repository

Summary

• RAG evaluation measures retrieval (context recall, precision) and generation (faithfulness, relevance) separately. - Faithfulness ≥ 0.85 is the minimum bar for production RAG - unfaithful answers are hallucinations. - Fix retrieval (context recall) before optimizing generation - the LLM can't faithfully answer without the right context. - Use failure stage breakdown to diagnose whether retrieval or generation caused each failure.

• Automate RAG eval in CI with RAGAS or custom scorers; calibrate against human labels. - End-to-end correctness alone is insufficient - a lucky hallucination passes correctness but fails faithfulness.