AI Evaluation Guide

TL;DR

• AI evaluation is your regression suite for non-deterministic systems. Traditional software has compilers, types, and unit tests. LLM pipelines have none of that by default — evaluation is the measurement layer you build to make changes safe.

• Evaluation spans four layers: retrieval (did we find the right documents?), generation (is the answer correct and faithful?), RAG end-to-end (does the full pipeline work?), and prompt (did the template change break behavior?). Each layer fails independently and needs its own metrics.

• Build a golden set of 50–200 real production queries with expected answers, relevance labels, or scoring rubrics. Version it in git. It is the single highest-leverage artifact in your AI stack.

• Run evals in CI and block deploys on regression. A prompt tweak that "felt fine" in staging is exactly the change that quietly drops faithfulness 8%. Manual spot-checking does not survive deadline pressure.

• Public benchmarks measure model capability; product eval measures your system. MMLU scores do not predict how a model handles your refund-policy taxonomy. Use benchmarks to shortlist models, then decide with your own eval.

On this page

• Why This Matters
• The Problem AI Evaluation Solves
• How We Got Here
• What Is AI Evaluation?
• How AI Evaluation Works
• Architecture
• Step-by-Step Flow
• Real Production Example
• Design Decisions
• Comparisons
• Common Mistakes
• Where It Breaks Down
• When NOT to Trust Your Eval Numbers
• Running in Production
• Related Guides
• Interview Questions
• Key Takeaways
• FAQs
• References
• Further Reading

Why This Matters

You shipped a RAG chatbot. Last week's prompt tweak "felt fine" in staging. This week, support tickets spike — the bot confidently cites an outdated refund policy. Your PM asks what changed and how bad it is. You have no number, no baseline, and no way to tell whether the regression came from the prompt, the retriever, or a silent model version bump from your provider.

This is the default state of AI systems without evaluation. LLM outputs are non-deterministic, subjective, and multi-dimensional. "Does it work?" is not binary, and the failure modes are silent: an HTTP 200 with a hallucinated answer looks identical to an HTTP 200 with a correct one. The disciplines that make traditional software safe to change — type checking, unit tests, integration tests — have no automatic equivalent for a pipeline whose core component is a probability distribution over tokens.

Evaluation is what turns AI development from artisan craft into engineering. It gives you regression detection before deploy, data-driven model selection, quality gates that survive deadline pressure, and numbers you can put in front of stakeholders. Teams that skip it iterate slower, not faster: every change becomes a coin flip, every incident becomes archaeology, and every model migration becomes a leap of faith. Teams that invest in eval ship faster because they trust their changes — the same reason teams with good test suites refactor fearlessly.

This guide is the hub for the evaluation cluster. It covers the layered model, golden sets, scorer selection, and CI gates; the four child guides — LLM Evaluation, RAG Evaluation, Retrieval Evaluation, and Prompt Evaluation — go deep on each layer. For multi-step tool-using systems, continue to Agent Evaluation.

The Problem AI Evaluation Solves

The core problem: how do you make changes to a non-deterministic system without breaking it silently?

Every production LLM application is a pipeline — query rewriting, retrieval, reranking, prompting, generation, post-processing — and each stage can regress independently. Without systematic measurement:

• Prompt changes cause silent regressions. Rewording an instruction shifts output distribution in ways no reviewer catches by reading five examples.
• Model upgrades improve benchmarks but hurt your use case. A new model version scores higher on public leaderboards and worse on your domain's citation formatting.
• Retrieval failures masquerade as generation failures. The model "hallucinated" — because the right document never entered the context window. You tune prompts for a week when the bug was recall@5.
• You discover problems from user complaints, not before deploy. By the time thumbs-down rates move, thousands of bad answers shipped.
• You cannot compare models on your tasks. Choosing between a Claude Sonnet 5 and a GPT-5.6 Terra deployment based on vendor benchmark tables is choosing blind — the only comparison that matters runs on your golden set.

Evaluation solves this by inserting a measurement layer between your application and its probabilistic components — the same role integration tests play for traditional services, adapted for outputs where correctness is graded, not binary.

How We Got Here

Evaluation practice lagged capability by years. Teams shipped impressive demos, then discovered that "impressive in a demo" and "reliable at the p95 of real traffic" are different properties.

Diagram: The evolution of AI evaluation practice

timeline
    title From vibes to CI gates
    2020-2022 : Academic benchmarks dominate
              : GLUE, MMLU as proxy for quality
    2022-2023 : Chat products ship on vibes
              : Manual spot-checks, demo-driven QA
    2023-2024 : Eval frameworks emerge
              : RAGAS, DeepEval, LLM-as-judge
    2024-2025 : Platform consolidation
              : Braintrust, LangSmith datasets, CI hooks
    2025-2026 : Eval as deploy gate
              : Golden sets in git, regression thresholds block merges

Each era's tooling answered the previous era's incident reports: benchmarks didn't predict product quality, vibes didn't survive scale, and ad-hoc scripts didn't survive team turnover.

Three shifts defined the current practice:

1. From benchmarks to product eval. Early teams selected models by leaderboard and assumed quality transferred. It didn't — public benchmarks measure broad capability on academic distributions, not your users' queries, your documents, or your output format. Benchmarks became a shortlisting tool; golden sets became the decision tool. See Benchmarks.

2. From end-to-end scores to layered metrics. A single "answer accuracy" number cannot tell you whether to fix retrieval or generation. The field converged on decomposition: retrieval metrics (recall@k, MRR, nDCG), generation metrics (faithfulness, correctness), and end-to-end metrics — each with a different owner and a different fix.

3. From manual review to automated gates. LLM-as-judge made subjective scoring cheap enough to run on every pull request. Combined with golden sets versioned in git, evaluation became a CI stage — the same trust boundary that unit tests provide for deterministic code.

What Is AI Evaluation?

AI evaluation is the practice of measuring an AI system's quality against defined criteria, using automated metrics, LLM-as-judge scoring, and human review on a curated, versioned test set. It is not a single score — it is a stack of measurements aligned to failure modes, so that when a number drops, you know which component broke and who owns the fix.

The four layers

Engineering Insight

Most production failures trace to retrieval, not generation. If recall@5 is 60%, no amount of prompt engineering reaches 95% answer accuracy — the evidence simply isn't in the context window. Always measure bottom-up.

The golden set

The golden set is the core artifact: a curated collection of test cases drawn from real production traffic, each carrying the inputs and labels the scorers need.

A useful golden-set case includes:

• The query — real user phrasing, including typos and ambiguity, not sanitized synthetic questions
• Relevance labels — which document or chunk IDs should be retrieved (for retrieval scoring)
• Expected answer or key facts — an exact answer, required phrases, or facts the answer must contain
• A rubric — for subjective cases, the criteria a judge should score against
• Severity metadata — compliance-critical cases get zero-tolerance treatment; ordinary cases contribute to averages

Fifty cases is a workable start for CI; 150–200 gives you confidence for model selection and quarterly audits. The set lives in git, changes go through review (changing an expected answer is changing the spec), and cases rotate in from production failures so the set tracks real usage instead of rotting.

Benchmarks vs product evaluation

These are different instruments answering different questions:

• Public benchmarks (MMLU, HELM, MT-Bench, SWE-bench and successors) measure model capability across broad academic distributions. They are useful for shortlisting: a model that cannot pass basic reasoning benchmarks will not suddenly excel at your tasks.
• Product evaluation measures your system — model plus retrieval plus prompts plus post-processing — on your distribution. It is the only measurement that predicts what users experience.

The failure pattern to avoid: treating a leaderboard delta as a deploy decision. A model two points higher on MMLU can be ten points worse on your citation-format compliance. Benchmarks are inputs to a shortlist; your golden set makes the call. Full treatment in Benchmarks.

How AI Evaluation Works

A production eval system has three moving parts: test data (the golden set), scorers (how each output gets graded), and orchestration (when evals run and what they gate).

Scorer types

When to use automated checks vs LLM-as-judge vs human review

This is the decision teams get wrong most often — defaulting to a judge for everything (expensive, noisy) or rules for everything (blind to quality).

Judge model selection is a workload decision, not a loyalty decision. Use a fast, cheap model (Claude Haiku 4.5, GPT-5.6 Luna, or a Gemini 3.7 Flash-tier deployment) for high-volume screening judgments, and reserve a frontier model (Claude Opus 4.8, GPT-5.6 Sol) for disputed cases, meta-evaluation, and rubric-heavy scoring. Critically, never use the same model as generator and judge without checking self-preference bias — models rate their own family's outputs systematically higher.

Production Tip

Calibrate every judge against 50–100 human-labeled cases before trusting it, and re-check quarterly. An uncalibrated judge is a random-number generator with an invoice.

Orchestration

Evals run at three cadences, each with a different scope:

1. Per-PR smoke suite (~20–50 cases, < 5 minutes): runs on every change to prompts, retrieval config, or model IDs. Gates the merge.
2. Nightly full suite (150+ cases): full metrics across all layers, trend lines, drift detection against a pinned baseline.
3. Ad-hoc deep runs: model migrations, embedding upgrades, chunking changes — the expensive comparisons you run a few times a quarter.

Architecture

A scalable eval architecture separates the golden set, the runner, the scorers, and the gate — so each can evolve independently.

Diagram: Production evaluation architecture

flowchart TB
    subgraph Data [Test data]
        GS[Golden set in git\nqueries + labels + rubrics]
        PF[Production feedback\nthumbs-down, escalations]
    end
    subgraph Exec [Execution]
        Runner[Eval runner\nCI job or scheduler]
        Pipe[System under test\nretrieval + generation]
    end
    subgraph Score [Scoring]
        Rules[Rule-based checks]
        RMet[Retrieval metrics\nrecall@k, MRR]
        Judge[LLM-as-judge\nfaithfulness, rubrics]
    end
    subgraph Decide [Decision]
        Store[(Results store\nhistorical scores)]
        Gate{CI gate\ndelta vs baseline}
        Dash[Dashboards + alerts]
    end
    GS --> Runner --> Pipe --> Rules & RMet & Judge
    Rules & RMet & Judge --> Store --> Gate
    Store --> Dash
    Gate -->|pass| Deploy[Merge / deploy]
    Gate -->|fail| Block[Block + report failing cases]
    PF -->|weekly triage| GS

The golden set feeds the runner; scorers feed a results store; the gate compares deltas against a baseline. Production failures flow back into the golden set so the suite tracks reality.

Two architectural rules matter more than tool choice:

Evaluate the real pipeline. The runner must call the same retrieval, the same prompt assembly, and the same model configuration as production. Evaluating a simplified replica measures the replica.

Compare deltas, not absolutes. A faithfulness of 0.87 means nothing in isolation; a drop from 0.91 to 0.87 after a prompt change means everything. Pin a baseline snapshot and gate on movement.

Production Tip

Store retrieval logs (chunk IDs, scores) alongside generation outputs for every eval case. When faithfulness drops, you need to know whether retrieval or generation failed — not guess. This is the same data your observability stack captures in production; eval and observability should share schemas.

Step-by-Step Flow

The flow below is the CI gate path — the highest-value eval loop, run on every pull request that touches prompts, retrieval, or model configuration.

Diagram: CI eval gate on a prompt change

sequenceDiagram
    participant Dev as Engineer
    participant CI as CI pipeline
    participant Run as Eval runner
    participant Sys as System under test
    participant J as Scorers (rules + judge)
    participant Base as Baseline store
    Dev->>CI: PR changes answer prompt v14 → v15
    CI->>Run: Trigger smoke suite (40 cases)
    loop each golden case
        Run->>Sys: Execute full pipeline
        Sys-->>Run: Answer + retrieved chunks + metadata
        Run->>J: Score retrieval, faithfulness, format
        J-->>Run: Per-case scores
    end
    Run->>Base: Fetch baseline (prompt v14 snapshot)
    Base-->>Run: recall@5 0.88, faithfulness 0.91
    alt deltas within thresholds and zero critical failures
        Run-->>CI: PASS — report deltas
        CI-->>Dev: Merge allowed
    else regression detected
        Run-->>CI: FAIL — faithfulness 0.91 → 0.83
        CI-->>Dev: Merge blocked + failing case IDs
    end

The gate compares against a pinned baseline and reports failing case IDs, so the engineer debugs specific regressions instead of staring at an aggregate.

1. Curate the golden set. Export real production queries (anonymized). Label relevant doc IDs, expected answers or key facts, and rubrics. Mark compliance-critical cases. Commit to git.
2. Pin a baseline. Run the full suite against the current production configuration and store the scores as the reference snapshot.
3. Trigger on change. CI runs the smoke suite on every PR touching prompts, retrieval config, chunking, embedding models, or model IDs.
4. Execute the real pipeline. Each case runs through production code paths — same retrieval, same prompt assembly, temperature 0 (or averaged over 3 runs where variance is unavoidable).
5. Score each layer. Retrieval metrics from labeled doc IDs; rule-based format checks; judge-scored faithfulness and correctness with rubrics.
6. Compare deltas. Aggregate per-metric; compare against baseline with tolerance thresholds (e.g., recall@5 within 2%, faithfulness within 3%). Any critical-severity case failure is an automatic block.
7. Gate the merge. Pass → merge with the delta report attached. Fail → block with failing case IDs and per-case diffs.
8. Refresh the loop. Weekly, triage production thumbs-downs and escalations; promote the best failures into the golden set; retire stale cases. Update the baseline when intentional improvements land.

Real Production Example

A support-automation team runs a RAG assistant over their policy corpus. An engineer edits the answer prompt to make responses "more concise." In staging, ten hand-checked answers look great. The CI eval gate tells a different story: the shortened prompt dropped the instruction that pinned answers to retrieved context, and faithfulness collapsed on cases where retrieval was noisy.

The gate implementation:

from __future__ import annotations

import json
import sys
from dataclasses import dataclass


@dataclass
class EvalCase:
    case_id: str
    query: str
    relevant_doc_ids: list[str]
    required_facts: list[str]
    severity: str  # "critical" | "normal"
    rubric: str | None = None


@dataclass
class CaseResult:
    case_id: str
    recall_at_5: float
    faithfulness: float
    facts_present: bool
    severity: str


THRESHOLDS = {
    "recall_at_5_max_drop": 0.02,
    "faithfulness_max_drop": 0.03,
    "min_pass_rate": 0.90,
}


class EvalGate:
    def __init__(self, pipeline, judge, baseline_path: str):
        self.pipeline = pipeline      # real production pipeline
        self.judge = judge            # calibrated judge client
        self.baseline = json.load(open(baseline_path))

    def score_retrieval(self, retrieved: list[str], relevant: list[str]) -> float:
        if not relevant:
            return 1.0
        return len(set(retrieved[:5]) & set(relevant)) / len(relevant)

    def score_faithfulness(self, answer: str, context: str) -> float:
        # Claim decomposition + entailment via a cheap screening judge
        # (Haiku 4.5 / GPT-5.6 Luna class); disputed cases re-scored
        # by a frontier judge offline.
        return self.judge.faithfulness(answer=answer, context=context)

    def run_case(self, case: EvalCase) -> CaseResult:
        out = self.pipeline.query(case.query, temperature=0)
        retrieved_ids = [c.id for c in out.chunks]
        context = "\n".join(c.text for c in out.chunks)
        return CaseResult(
            case_id=case.case_id,
            recall_at_5=self.score_retrieval(retrieved_ids, case.relevant_doc_ids),
            faithfulness=self.score_faithfulness(out.answer, context),
            facts_present=all(
                f.lower() in out.answer.lower() for f in case.required_facts
            ),
            severity=case.severity,
        )

    def run(self, cases: list[EvalCase]) -> int:
        results = [self.run_case(c) for c in cases]
        recall = sum(r.recall_at_5 for r in results) / len(results)
        faith = sum(r.faithfulness for r in results) / len(results)
        pass_rate = sum(r.facts_present for r in results) / len(results)
        critical_failures = [
            r.case_id for r in results
            if r.severity == "critical" and (not r.facts_present or r.faithfulness < 0.9)
        ]

        failures = []
        if self.baseline["recall_at_5"] - recall > THRESHOLDS["recall_at_5_max_drop"]:
            failures.append(f"recall@5 {self.baseline['recall_at_5']:.2f} -> {recall:.2f}")
        if self.baseline["faithfulness"] - faith > THRESHOLDS["faithfulness_max_drop"]:
            failures.append(f"faithfulness {self.baseline['faithfulness']:.2f} -> {faith:.2f}")
        if pass_rate < THRESHOLDS["min_pass_rate"]:
            failures.append(f"pass rate {pass_rate:.2f} < {THRESHOLDS['min_pass_rate']}")
        if critical_failures:
            failures.append(f"critical case failures: {critical_failures}")

        report = {
            "recall_at_5": recall, "faithfulness": faith,
            "pass_rate": pass_rate, "failures": failures,
        }
        print(json.dumps(report, indent=2))
        return 1 if failures else 0


if __name__ == "__main__":
    sys.exit(EvalGate(pipeline, judge, "baselines/prod.json").run(load_golden_set()))

Wired into CI:

# .github/workflows/eval-gate.yml
on:
  pull_request:
    paths: ['prompts/**', 'retrieval/**', 'config/models.yml']
jobs:
  eval-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: python evals/run_gate.py --suite smoke
        # non-zero exit blocks the merge

The outcome for the "more concise" PR: the gate failed with faithfulness 0.91 -> 0.83 and three critical case IDs. The engineer inspected the failing cases, saw the model padding gaps with parametric guesses on weak-retrieval queries, restored the grounding instruction while keeping the brevity changes, and the re-run passed with faithfulness at 0.90. Total cost: one CI cycle. Without the gate: a week of degraded answers, a support-ticket spike, and an incident review.

Design Decisions

Common patterns

Decision matrix

When should I invest in which layer first?

Comparisons

Common Mistakes

1. Evaluating only end-to-end accuracy. A correct answer can mask lucky retrieval; a wrong one doesn't tell you which stage failed. Decompose into layered metrics with separate owners.

2. Using leaderboard benchmarks for product decisions. MMLU deltas do not predict performance on your support-ticket taxonomy. Shortlist with benchmarks; decide with your golden set.

3. Golden sets that don't match production. Synthetic questions like "What is our refund policy?" miss real phrasing, typos, multi-intent queries, and adversarial inputs. Export from real traffic.

4. Evaluating only the happy path. A suite of well-formed, answerable questions tells you nothing about behavior on out-of-scope queries, empty retrieval, ambiguous intent, or injection attempts. Failure cases belong in the golden set.

5. LLM-as-judge without calibration. Judges are systematically lenient, verbose-answer-biased, and self-preferring. Calibrate against 50–100 human labels; re-check quarterly.

6. No version control on test cases. Changing an expected answer without review silently redefines "passing" and invalidates every trend line.

7. Skipping eval in CI. "We'll test manually before release" fails every sprint under deadline pressure. Automate the gate or accept regressions as a business cost.

8. Gating on absolutes instead of deltas. "Faithfulness must exceed 0.85" passes a regression from 0.95 to 0.86. Gate on movement against a pinned baseline.

Common Mistake

Optimizing a single metric while ignoring its complement. Maximizing faithfulness alone produces a system that refuses rather than answers — users experience "the bot is broken." Track refusal rate and answer rate alongside quality metrics.

Where It Breaks Down

• Non-reproducible outputs. Temperature > 0, provider-side model updates, and cache effects introduce variance that can exceed your gate thresholds. Run evals at temperature 0 where possible; average 3 runs where not; pin model versions and alert when providers deprecate them.

• Label quality ceilings. Your metrics are only as good as your labels. Garbage expected answers produce garbage pass rates that everyone trusts anyway. Budget for label review, not just label creation.

• Eval set contamination. If golden queries leak into few-shot examples, fine-tuning data, or the retrieval index in ways production queries wouldn't, metrics inflate. Hold out test cases and audit for leakage after data pipeline changes.

• Metric gaming. Once a metric gates deploys, teams optimize the metric. Prompts evolve to please the judge rather than the user. Rotate fresh cases from production quarterly and meta-evaluate judge-human agreement.

• Judge drift. Judge model upgrades silently shift scoring distributions, making trend lines lie. Pin judge model versions; re-baseline explicitly when you upgrade the judge.

• Cost and latency at scale. A 200-case suite with judge scoring costs real money per run and can take 30+ minutes. Tier aggressively: rules first (free), retrieval metrics second (cheap), judges last (sampled where needed).

• Multi-turn and agentic systems. Single-shot Q&A eval doesn't capture conversation state, tool-call sequencing, or error recovery. You need scripted conversation cases and trajectory-level scoring — harder to label, harder to gate.

When NOT to Trust Your Eval Numbers

Evaluation is necessary, but there are situations where the numbers you have actively mislead — and the correct move is to distrust them:

• When your "eval" is a public benchmark. A leaderboard position is not product readiness. A model can top reasoning benchmarks and fail your citation format, your language mix, or your latency budget. If your launch decision cites MMLU and not a golden set, you have not evaluated your product.

• When the suite only covers the happy path. A 98% pass rate on answerable, well-formed questions says nothing about the 20% of real traffic that is out-of-scope, ambiguous, adversarial, or hits empty retrieval. Report coverage alongside scores.

• When the golden set has rotted. If cases haven't rotated in from production for two quarters, your suite measures last year's product. Treat golden-set staleness like dependency staleness.

• When the judge is uncalibrated or upgraded silently. Scores from a judge that no human has audited — or whose underlying model changed mid-trend-line — are not measurements.

• When offline metrics and user signal diverge. If eval says quality improved and thumbs-down rates say otherwise, believe the users and fix the eval. Offline eval is a proxy; production feedback is the referent.

• When variance exceeds the effect size. A 2% faithfulness delta on a 40-case suite with sampling variance is noise. Don't ship — or block — on it. Increase the sample or reduce the variance first.

Warning

The most expensive eval failure is false confidence: a green dashboard built on a stale set, an uncalibrated judge, and happy-path cases. It fails precisely when you need it — during the incident.

Running in Production

Best Practice

Publish your golden-set metrics internally and block deploys on regression. A quality number nobody sees is a quality number nobody defends.

Production checklist

• Golden set (50+ cases) versioned in git with an owner and review process
• Real production queries, including failure cases and out-of-scope inputs
• Retrieval labels (relevant doc IDs) for every RAG case
• Layered metrics: recall@k, faithfulness, answer correctness, format compliance
• Severity tiers — compliance-critical cases block on any failure
• CI smoke suite on every prompt/retrieval/model PR with a hard gate
• Baseline snapshot stored; gates compare deltas, not absolutes
• Judge calibrated against human labels; agreement re-checked quarterly
• Weekly triage promotes production failures into the golden set
• Retrieval logs captured per eval case for failure attribution
• Dashboard with trend lines reviewed with stakeholders

Related Guides

Diagram: The evaluation learning path

flowchart LR
    LLM[Large Language Models] --> EV[AI Evaluation - you are here]
    RAG[RAG] --> EV
    EV --> RE[Retrieval Evaluation]
    EV --> LE[LLM Evaluation]
    EV --> RGE[RAG Evaluation]
    EV --> PE[Prompt Evaluation]
    EV --> AE[Agent Eval]
    EV --> BM[Benchmarks]
    RE & LE & RGE --> OBS[Observability]
    LE --> HD[Hallucination Detection]
    HD --> GR[Guardrails]

Start at the hub, go deep per layer, then connect eval to runtime: observability feeds the golden set, detection and guardrails enforce per-request what eval verifies in aggregate.

The evaluation cluster:

• LLM Evaluation — output quality, LLM-as-judge mechanics, golden sets for generation
• RAG Evaluation — end-to-end RAG metrics: faithfulness, context precision, answer correctness
• Retrieval Evaluation — recall@k, MRR, nDCG, and building relevance labels
• Prompt Evaluation — regression testing for templates and few-shot examples
• Agent Evaluation — trajectories, tool-call accuracy, and task success for agent loops
• Benchmarks — public leaderboards: what they measure and what they don't

Runtime counterparts:

• Observability — traces and metrics that feed eval datasets and carry eval scores as alerts
• Hallucination Detection — per-request faithfulness verification at runtime
• Guardrails — policy enforcement on individual outputs

Tools: LangChain · LlamaIndex

Interview Questions

1. Why can't you test LLM systems the way you test traditional software?
   Outputs are non-deterministic, correctness is graded rather than binary, and quality depends on multiple pipeline stages (retrieval, prompting, generation) that fail independently. You replace assertions with metrics on a curated test set and gate on statistical deltas rather than exact matches.

2. What are the four layers of AI evaluation and why decompose?
   Retrieval (recall@k, MRR), generation (faithfulness, correctness), RAG end-to-end (answer quality), and prompt (regression deltas). Decomposition tells you which component broke: an end-to-end score drop can't distinguish a retrieval miss from an unfaithful generator, and the fixes have different owners.

3. What makes a good golden set?
   Real production queries (not synthetic), relevance labels and expected facts, rubrics for subjective cases, severity tiers, failure and out-of-scope cases alongside happy paths, version control with review, and a refresh loop from production failures.

4. When would you use LLM-as-judge vs rules vs human review?
   Rules for anything verifiable (format, required facts, schema) — deterministic and free. Judge for subjective, high-volume scoring (faithfulness, helpfulness) with explicit rubrics. Humans for calibration, disputes, and compliance-critical cases. The judge must be calibrated against human labels before it counts.

5. What are the known biases of LLM-as-judge?
   Leniency, verbosity bias (longer answers score higher), position bias in pairwise comparison, and self-preference (rating its own model family higher). Mitigate with rubrics, randomized ordering, a different model family as judge, and periodic human calibration.

6. Why are public benchmarks insufficient for product decisions?
   They measure broad capability on academic distributions — not your domain, format constraints, language mix, or cost/latency envelope. They're useful for shortlisting models; the deploy decision needs your golden set.

7. How would you design a CI eval gate?
   Smoke suite of 20–50 cases triggered on prompt/retrieval/model changes; execute the real pipeline at temperature 0; score each layer; compare deltas against a pinned baseline (e.g., recall@5 within 2%, faithfulness within 3%); zero tolerance on critical-severity cases; fail the build with failing case IDs.

8. How do you handle output variance in evals?
   Temperature 0 where the product allows; otherwise average over 3+ runs per case. Pin model versions. Ensure gate thresholds exceed measured run-to-run variance, or you'll block PRs on noise.

9. A user reports wrong answers but your eval dashboard is green. What do you check?
   Golden-set coverage (does the set include this query type?), set staleness, judge calibration drift, eval-production config divergence (different model version or retrieval index), and contamination. Then promote the failing queries into the golden set.

10. How do evaluation and observability relate?
    Observability records what happened per request — traces, retrieval context, costs. Evaluation measures whether outputs were good against ground truth. Traces feed golden sets; eval scores become monitored metrics with alerts. Neither substitutes for the other.

11. How do you evaluate a model migration, e.g., moving a workload from a Sonnet-class to a GPT-5.6 Terra-class model?
    Run the full golden set against both configurations with identical retrieval and prompts, compare per-layer metrics and per-case diffs (not just aggregates), check format compliance and refusal-rate shifts specifically, and canary in production with quality-proxy monitoring before full cutover.

12. What is metric gaming and how do you defend against it?
    Once a metric gates deploys, optimization pressure targets the metric — prompts that please the judge rather than users. Defend with quarterly case rotation from production, judge-human agreement audits, multiple complementary metrics, and divergence alerts between offline scores and user feedback.

Key Takeaways

• Evaluation is the regression suite for non-deterministic systems — without it, every change is a coin flip discovered by users.
• Decompose into four layers (retrieval, generation, end-to-end, prompt); measure bottom-up because retrieval failures masquerade as generation failures.
• The golden set is the core artifact: real queries, versioned in git, refreshed from production failures, with severity tiers.
• Choose scorers by verifiability: rules for facts and format, calibrated judges for subjective quality, humans for calibration and critical cases.
• Gate deploys on deltas against a pinned baseline in CI; absolutes and manual spot-checks don't survive deadline pressure.
• Benchmarks shortlist models; your golden set makes decisions. Never confuse leaderboard position with product readiness.

FAQs

What's the minimum viable eval setup?

Fifty real queries with expected key facts, recall@5 for retrieval, a calibrated faithfulness check, and a CI job that runs on every prompt change and fails on regression. Expand from there.

How is AI evaluation different from traditional ML evaluation?

Classical ML evaluates a single model on fixed inputs with labels. LLM systems are pipelines (retrieve → assemble → generate → post-process) with subjective, variable outputs. You evaluate layers separately, tolerate variance, and rely on graded scoring rather than exact accuracy.

Should I use LLM-as-judge or human review?

Both, in a specific relationship: humans label 50–100 cases to calibrate the judge, the judge scales to full-suite scoring, and humans re-audit agreement quarterly plus handle disputed and critical cases.

Which model should I use as a judge?

A fast-tier model (Claude Haiku 4.5, GPT-5.6 Luna, Gemini 3.7 Flash-class) for volume screening; a frontier model (Claude Opus 4.8, GPT-5.6 Sol) for disputes and meta-eval. Prefer a different model family than your generator to reduce self-preference bias, and pin the judge version.

How often should I run evals?

Smoke suite (20–50 cases) on every relevant PR. Full suite (150+ cases) nightly or before releases. Deep comparative runs when changing models, embeddings, or chunking.

What metrics matter most for RAG?

Retrieval recall@k first — it caps everything downstream. Then faithfulness, then end-to-end answer correctness. Fixing retrieval has the highest ROI. See RAG Evaluation.

How do I build a golden test set?

Export anonymized production queries, prioritizing failures from support tickets and thumbs-downs. Have domain experts write expected answers and relevance labels. Include out-of-scope and adversarial cases. Version in git; refresh quarterly.

Can I evaluate without labeled data?

Partially. LLM-as-judge with rubrics, consistency checks (same query, multiple runs), and reference-free faithfulness (answer vs retrieved context) work without answer labels. Retrieval metrics still need relevance labels — synthetic labels from chunk metadata can bootstrap them.

What thresholds should block a deploy?

Common defaults: no more than a 2% drop in recall@5, no more than a 3% drop in faithfulness, and zero failures on critical compliance cases. Set thresholds above your measured run-to-run variance or you'll block on noise.

How do I evaluate multi-turn conversations?

Store scripted conversations in the golden set. Score each turn's retrieval and the final answer, plus conversation-level properties (context carry-over, contradiction). Trajectory-level scoring for agents adds tool-call correctness and recovery behavior.

Does evaluation slow down development?

Setup takes days; after that it accelerates development the way test suites do — regressions surface in CI instead of production, and engineers change prompts without fear. Teams without eval spend the saved time firefighting.

Is a high benchmark score enough to switch models?

No. Benchmarks shortlist candidates. Run your golden set against the new model with identical retrieval and prompts, inspect per-case diffs, and canary before cutover.

References

• RAGAS Documentation
• DeepEval Documentation
• OpenAI Evals — Best Practices
• Anthropic — Evaluating AI Systems
• LangSmith Evaluation Documentation

Further Reading

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al., 2023)
• Holistic Evaluation of Language Models — HELM (Liang et al., 2022)
• Braintrust — Eval Best Practices
• Arize Phoenix Documentation