Evaluation Guide

TL;DR

• AI evaluation measures whether your system meets requirements - not whether the model tops a leaderboard. Your golden set, your rubrics, your thresholds.

• Evaluation spans four layers: retrieval (did we find the right docs?), generation (is the answer correct and faithful?), prompt (did the template change break behavior?), and end-to-end (does the user get what they need?).

• Build a golden test set of 50–200 real production queries with expected answers or scoring rubrics. This is your regression suite for every model, prompt, or pipeline change.

• Run evals in CI - block deploys that drop metrics below threshold. Manual spot-checking does not survive deadline pressure.

• Separate metrics by failure mode - a 5% drop in retrieval recall and a 5% drop in faithfulness require different fixes and different owners.

Why This Matters

You shipped a RAG chatbot. Last week's prompt tweak "felt fine" in staging. This week, support tickets spike - the bot now confidently cites outdated refund policies. Your PM asks for a number. You have no number.

Without evaluation, every change to prompts, models, chunking, or retrieval is a coin flip. LLM outputs are non-deterministic, subjective, and multi-dimensional. "Does it work?" is not binary. Evaluation turns AI development from artisan craft into engineering: regression detection, model selection data, quality gates, and stakeholder communication backed by metrics.

Teams that skip evaluation iterate slower, ship more regressions, and cannot justify model spend. Teams that invest in eval ship faster because they trust their changes.

The Problem AI Evaluation Solves

Traditional software has compilers, type checkers, and unit tests. AI systems have none of that by default. The same input can produce different outputs. Quality depends on retrieval, prompting, model choice, and post-processing - each layer can fail independently.

Without systematic evaluation:

• Prompt changes cause silent regressions
• Model upgrades improve benchmarks but hurt your specific use case
• Retrieval failures masquerade as generation failures
• You discover problems from user complaints, not before deploy
• You cannot set SLAs or compare GPT-4o vs Claude on your tasks

AI evaluation provides a measurement layer between your application and probabilistic models - the same role integration tests play for traditional software.

What Is AI Evaluation?

AI evaluation is the practice of measuring system quality against defined criteria using automated metrics, LLM-as-judge scoring, and human review on a curated test set. It is not a single score - it is a stack of measurements aligned to failure modes:

Engineering Insight

Most production failures trace to retrieval, not generation. If recall@5 is 60%, no prompt engineering reaches 95% answer accuracy. Always measure bottom-up.

How AI Evaluation Works

A production eval system has three components: test data, scorers, and orchestration.

RAG couples a dense vector index of external knowledge with a sequence-to-sequence generator. At query time, the retriever selects relevant passages and the generator conditions its answer on that evidence.

Evaluation Methods

Architecture

A scalable eval architecture separates concerns:

A production RAG stack separates offline indexing (load, chunk, embed, store) from online querying (embed query, retrieve, rerank, generate).

Production Tip

Store retrieval logs (chunk IDs, scores) alongside generation outputs. When faithfulness drops, you need to know whether retrieval or generation failed - not guess.

Real Production Example

A legal-tech RAG system runs three eval layers on every PR:

from dataclasses import dataclass
from typing import Optional
import json

@dataclass
class EvalCase:
    query: str
    relevant_doc_ids: list[str]
    expected_answer_contains: list[str]  # key phrases
    rubric: Optional[str] = None

@dataclass
class EvalResult:
    case_id: str
    recall_at_5: float
    faithfulness: float
    answer_score: float
    passed: bool

class ProductionEvalSuite:
    THRESHOLDS = {
        "recall_at_5": 0.85,
        "faithfulness": 0.90,
        "answer_score": 0.80,
    }

    def __init__(self, pipeline, judge_model="gpt-4o"):
        self.pipeline = pipeline
        self.judge = judge_model

    def score_retrieval(self, retrieved_ids: list[str], relevant_ids: list[str], k: int = 5) -> float:
        top_k = set(retrieved_ids[:k])
        relevant = set(relevant_ids)
        if not relevant:
            return 1.0
        return len(top_k & relevant) / len(relevant)

    def score_faithfulness(self, answer: str, context: str) -> float:
        # Simplified - production uses NLI or LLM-as-judge
        prompt = f"""Rate faithfulness 0-1. Does the answer ONLY use facts from context?
Context: {context[:4000]}
Answer: {answer}
Return JSON: {{"score": float}}"""
        resp = self.pipeline.llm.generate(prompt, model=self.judge, temperature=0)
        return json.loads(resp)["score"]

    def run_case(self, case: EvalCase, case_id: str) -> EvalResult:
        result = self.pipeline.query(case.query)
        retrieved_ids = [c.id for c in result["chunks"]]
        context = "\n".join(c.text for c in result["chunks"])

        recall = self.score_retrieval(retrieved_ids, case.relevant_doc_ids)
        faith = self.score_faithfulness(result["answer"], context)
        answer_ok = all(phrase.lower() in result["answer"].lower()
                        for phrase in case.expected_answer_contains)

        passed = (
            recall >= self.THRESHOLDS["recall_at_5"]
            and faith >= self.THRESHOLDS["faithfulness"]
            and answer_ok
        )
        return EvalResult(case_id, recall, faith, float(answer_ok), passed)

def run_suite(self, cases: list[EvalCase]) -> dict:
        results = [self.run_case(c, f"case_{i}") for i, c in enumerate(cases)]
        avg_recall = sum(r.recall_at_5 for r in results) / len(results)
        avg_faith = sum(r.faithfulness for r in results) / len(results)
        pass_rate = sum(r.passed for r in results) / len(results)
        return {
            "recall_at_5": avg_recall,
            "faithfulness": avg_faith,
            "pass_rate": pass_rate,
            "passed": pass_rate >= 0.90 and avg_recall >= self.THRESHOLDS["recall_at_5"],
        }

The CI job blocks merge if recall_at_5 drops more than 2% from baseline or any individual high-severity case fails.

Decision Matrix

⚠ Common Mistakes

1. Evaluating only end-to-end accuracy. A correct answer can mask lucky retrieval. Decompose into retrieval + generation metrics.

2. Using leaderboard benchmarks for product decisions. MMLU scores do not predict how GPT-4o handles your support ticket taxonomy.

3. Golden sets that don't match production. Synthetic questions like "What is RAG?" miss real user phrasing, typos, and ambiguous intent.

4. No version control on test cases. Changing expected answers without review invalidates trend lines.

5. LLM-as-judge without calibration. A judge model can be systematically lenient. Sample 50 cases for human review monthly.

6. Skipping eval in CI. "We'll test manually" fails every sprint. Automate or accept regressions.

Common Mistake

Optimizing for a single metric (e.g., faithfulness) while ignoring recall creates systems that refuse to answer rather than hallucinate - users perceive this as "the bot is broken."

Where It Breaks Down

• Non-reproducible outputs - temperature > 0 introduces variance. Run evals at temperature 0 or average over 3 runs for critical cases.

• Label quality - garbage expected answers produce garbage metrics. Invest in label review.

• Eval set contamination - if golden queries appear in training data or index, metrics inflate. Hold out test cases from indexing.

• Metric gaming - teams optimize the eval, not the product.

Rotate fresh cases from production logs quarterly.

• Latency and cost - full eval suites with LLM-as-judge on 200 cases cost dollars per run. Tier: fast smoke (20 cases) on PR, full suite nightly.

Running AI Evaluation in Production

Best Practice

Log production queries that fail user feedback into a "candidate pool." Review weekly and promote the best cases into your golden set. Your eval set should evolve with real usage.

Ecosystem

Related Technologies

• LLM Evaluation: Output quality, LLM-as-judge, golden test sets for generation.

• RAG Evaluation: End-to-end RAG metrics - faithfulness, context utilization, answer correctness.

• Retrieval Evaluation: recall@k, MRR, nDCG for search quality.

• Prompt Evaluation: Regression testing for prompt templates and few-shot examples.

• Hallucination Detection: Runtime verification layer complementing offline eval.

• Benchmarks: Public leaderboards for model comparison - use as input, not substitute for your eval.

Learning Path

Prerequisites: Large Language Models · RAG

Next topics: LLM Evaluation · RAG Evaluation · Retrieval Evaluation · Prompt Evaluation

Estimated time: 50 min · Difficulty: Intermediate

FAQs

What's the minimum viable eval setup?

50 real queries, expected answers or key phrases, recall@5 for retrieval, and a faithfulness check. Run on every prompt change. Expand from there.

How is AI evaluation different from traditional ML evaluation?

ML models have fixed inputs and labels. LLM systems are pipelines (retrieve → generate) with subjective outputs. You evaluate layers separately and accept probabilistic variance.

Should I use LLM-as-judge or human review?

Both. Humans label 50–100 cases to calibrate the judge. Scale with the judge; spot-check humans monthly.

How often should I run evals?

Smoke eval (20 cases) on every PR. Full suite (150+ cases) nightly or before releases. Ad-hoc when changing models or embedding versions.

What metrics matter most for RAG?

Retrieval recall@k first, then faithfulness, then end-to-end answer correctness. Fixing retrieval has the highest ROI.

How do I build a golden test set?

Export real production queries (anonymized). Have domain experts write expected answers or relevance labels. Start with failure cases from support tickets.

Can I eval without labeled data?

Partially. Use LLM-as-judge with rubrics, consistency checks (same query, multiple runs), and production feedback. Labeled retrieval data still requires human doc IDs.

What threshold should block a deploy?

Depends on stakes. Common: no >2% drop in recall@5, no >3% drop in faithfulness, zero failures on critical compliance cases.

How do I eval multi-turn conversations?

Store conversation scripts in golden set. Eval each turn's retrieval and final answer. Track context carry-over separately.

Does eval slow down development?

Initial setup takes days. After that, eval accelerates development by catching regressions before review. Teams without eval spend more time firefighting.

References

• Hugging Face Transformers Documentation
• LangChain Documentation
• OpenAI API Documentation

Further Reading

• Anthropic Documentation
• Google AI for Developers
• LlamaIndex Documentation
• Pinecone Learning Center

Summary

• AI evaluation is a stack: retrieval → generation → end-to-end. Measure each layer.
• Build a golden test set from real production queries - 50 minimum, 200 for confidence.
• Run evals in CI with thresholds. Manual testing does not scale.
• Calibrate LLM-as-judge against human labels. Never trust a single metric.
• Evolve your test set from production failures. Static eval sets rot.
• Evaluation is the highest-leverage investment for production AI reliability.

Production Checklist

• Golden test set (50+ cases) versioned in git with ownership
• Retrieval labels (relevant doc IDs) for RAG queries
• Separate metrics: recall@k, faithfulness, answer correctness
• CI job runs smoke eval on every PR with pass/fail gate
• Baseline scores stored - compare deltas, not absolutes
• LLM-as-judge calibrated against human labels (sample monthly)
• Production feedback loop promotes new cases into golden set
• Retrieval logs captured for every eval case (debug failures)
• Critical compliance cases marked - zero tolerance for failure
• Dashboard with trend lines for weekly stakeholder review