LLM Evaluation Guide

TL;DR

• LLM evaluation measures whether outputs meet your requirements — correctness, format, safety, faithfulness, latency, cost — on your task distribution, not general "intelligence."

• A golden test set (50–200 real cases) with expected outputs or rubrics is the regression suite for every prompt, model, or pipeline change. Expand it from production failures.

• Combine scorers by failure mode: exact match / schema for structured paths; task validators (code exec, API checks); LLM-as-judge (pointwise and pairwise) for subjective quality; human labels for calibration.

• LLM-as-judge scales subjective scoring but is biased — position bias, verbosity bias, self-preference. Use a separate judge model, explicit rubrics, order swaps, and κ ≥ 0.7 vs humans before CI trust.

• Public benchmarks ≠ product quality. Benchmarks inform model selection; domain golden sets and evaluation hubs decide ship/no-ship. Gate deploys on regressions.

On this page

• Why This Matters
• The Problem LLM Evaluation Solves
• How We Got Here
• What Is LLM Evaluation?
• How LLM Evaluation Works
• Architecture
• Step-by-Step Flow
• Real Production Example
• Design Decisions
• Comparisons
• Common Mistakes
• Where It Breaks Down
• When NOT to Rely on Automated Eval Alone
• Running in Production
• Related Guides
• Interview Questions
• Key Takeaways
• FAQs
• References

Why This Matters

You shipped a support summarizer. After a "harmless" prompt tweak, users say answers feel worse. Your PM asks for a number. You have none — only anecdotes and a few cherry-picked demos.

Without evaluation, every change is a coin flip. Prompt edits, model upgrades (GPT-5.6 Sol vs Claude Sonnet 5 vs Gemini 3.5), temperature knobs, and context packing all alter behavior in ways that are invisible until a user complains — or until a compliance review finds invented policy text. LLM outputs are non-deterministic, multi-dimensional, and task-specific. "Does it work?" is not a boolean.

Evaluation turns LLM development into engineering:

1. Regression detection — know immediately when a change breaks a slice of the task
2. Model selection on your workload — compare families on your golden set, not MMLU alone
3. Quality gates — block deploys that fail thresholds
4. Debugging signal — failures localize to prompt, retrieval, model, or post-processing
5. Stakeholder communication — "pass rate 91% → 84%" is actionable; "it seems worse" is not

Teams that skip eval iterate slower, ship more silent regressions, and cannot justify model spend. Teams that invest ship faster because they trust changes.

The Problem LLM Evaluation Solves

LLMs are probabilistic generators. The same input can yield different outputs. Quality is subjective and task-dependent. There is no compiler for "correct summary," no built-in unit test framework, and no single accuracy number that covers format errors, factual errors, refusals, and latency regressions.

Without systematic evaluation you typically:

• Compare models by vibes or leaderboard scores that do not transfer to your domain
• Ship prompt changes that silently break edge cases
• Optimize for demo queries, not production distribution
• Discover failures from users, not CI
• Cannot set SLAs or quality targets with engineering rigor

LLM evaluation is the measurement layer between your application and the model — the role integration tests play for traditional software. Specialize further with prompt evaluation and RAG evaluation when those surfaces dominate.

How We Got Here

Eval practice lagged demos. Early teams shipped chat UIs with manual spot-checks. Scale forced automation; subjectivity forced judges; production forced CI.

Diagram: How LLM eval became a ship gate

timeline
    title From vibes to CI regression gates
    2020-2022 : Academic benchmarks
              : MMLU, HumanEval for capability
    2022-2023 : Chat products ship
              : Manual QA and demos
    2023-2024 : Golden sets + LLM-as-judge
              : Domain rubrics, RAGAS-style metrics
    2024-2026 : Eval in CI + online sampling
              : Calibrated judges, regression alerts

Capability leaderboards came first; product quality gates followed once silent regressions hit users.

The durable lesson: evaluate the system you ship (prompt + tools + retrieval + model) on a frozen distribution that mirrors production — then re-baseline when that distribution shifts.

What Is LLM Evaluation?

LLM evaluation is measuring model (or pipeline) output quality against defined criteria using:

The core artifact is a golden test set: curated inputs with expected outputs, rubrics, or pass/fail criteria that represent production task distribution.

# Minimal golden test case
{
    "id": "incident_047",
    "input": "Summarize this incident report: ...",
    "reference": "Server outage caused by DNS misconfiguration. Duration: 47 minutes.",
    "rubric": {
        "must_include": ["DNS", "47 minutes"],
        "must_not_include": ["customer data breach"],
        "format": "single paragraph, under 100 words",
        "faithfulness": "claims must be supported by the report"
    }
}

Evaluation runs your pipeline on every case, scores outputs, aggregates metrics, and compares against a versioned baseline.

Engineering Insight

Distinguish system eval (your app) from model eval (provider capability). Shipping decisions need system eval. Model cards and benchmarks are inputs to selection, not substitutes for golden sets.

How LLM Evaluation Works

Evaluation dimensions

Production systems need multi-dimensional scores:

• Correctness — factually right for the task; for RAG, grounded in context (hallucination detection)
• Format compliance — schema, markdown contract, required fields
• Completeness — all parts of the question addressed
• Safety — refuses harmful / out-of-policy requests; avoids PII leakage
• Consistency — similar inputs yield stable outputs at temperature 0
• Latency & cost — p50/p95 and tokens within budget

Exact match, rubrics, and validators

Start where failure is cheap to detect:

1. Parse JSON / apply JSON Schema
2. Required / forbidden substrings
3. Numeric equality with tolerances
4. Execute generated code or call a checker API

Use rubrics when answers are open-ended: score anchors ("0.5 means mostly correct with one missing entity") beat vague "rate helpfulness 1–5."

LLM-as-judge: pointwise and pairwise

Pointwise — one output, one rubric, one score (faithfulness 0–1, tone 1–5).

Pairwise — given input and two outputs, which is better? Useful for prompt A/B and model bake-offs. Always randomize order and average both orderings to reduce position bias.

JUDGE_PROMPT = """
You are an evaluation judge. Score faithfulness (0-1):
- 1.0: Every claim is supported by the provided context
- 0.5: Mostly supported; minor unsupported details
- 0.0: Contains claims not in the context

Context: {context}
Question: {question}
Response: {response}

Return JSON: {{"faithfulness": 0.0-1.0, "reasoning": "..."}}
"""

Judge rules that matter in production:

1. Use a different model family than the one under test when possible (avoid self-preference)
2. Prefer a capable judge for offline calibration (e.g., Claude Sonnet 5 or GPT-5.6 Sol) and a cheaper judge for CI (Haiku 4.5 / Gemini 3.7 Flash-class) once calibrated
3. Explicit rubrics with score anchors
4. Calibrate on 50+ human-labeled cases; target agreement κ ≥ 0.7
5. For pairwise, swap order and take majority / average
6. Temperature 0; optionally 3 votes for high-stakes gates

Human evaluation and calibration

Humans define ground truth for subjective dimensions and calibrate automated scorers. Weekly sample (e.g., 20 cases): dual-annotate, resolve disagreements, measure judge–human agreement, retire stale cases. Without this loop, CI optimizes for judge quirks.

Golden set design

CI and regression detection

Pin dataset version + prompt version + model ID. Store baseline metrics keyed by git SHA. Fail the build if pass rate or faithfulness drops beyond tolerance (e.g., >3% pass-rate drop). Online evaluation (sampled live traffic) complements offline golden sets but does not replace pre-deploy gates — see observability.

Diagram: Offline eval vs online monitoring

stateDiagram-v2
    [*] --> ChangeProposed: prompt/model/pipeline PR
    ChangeProposed --> OfflineEval: run golden set
    OfflineEval --> Blocked: regression vs baseline
    OfflineEval --> Deployed: gates pass
    Deployed --> OnlineSample: sample live traffic
    OnlineSample --> Alert: drift / user thumbs-down
    Alert --> ExpandGolden: add failing cases
    ExpandGolden --> ChangeProposed: next iteration
    Blocked --> ChangeProposed: fix and re-run

Offline golden sets block bad deploys; online sampling feeds the next golden-set expansion.

Architecture

A production LLM eval system has five components:

Diagram: LLM evaluation architecture

flowchart TB
    subgraph Inputs [Inputs]
        DS[Golden dataset vN]
        Pipe[System under test]
        Base[Baseline metrics]
    end
    subgraph Run [Eval run]
        Exec[Runner parallelize cases]
        Det[Deterministic scorers]
        Judge[LLM-as-judge]
        Hum[Human sample]
    end
    subgraph Out [Outputs]
        Agg[Aggregate + slice metrics]
        Gate{Regression gate}
        Rep[Report + artifacts]
    end
    DS --> Exec
    Pipe --> Exec
    Exec --> Det --> Agg
    Exec --> Judge --> Agg
    Hum -.->|calibrate| Judge
    Base --> Gate
    Agg --> Gate
    Gate -->|fail| Block[Block deploy]
    Gate -->|pass| Ship[Allow merge]
    Agg --> Rep

Dataset and baseline are versioned artifacts; judges are calibrated subordinates of human labels, not sources of truth.

Source: RAGAS Documentation

Step-by-Step Flow

Diagram: End-to-end eval run on a PR

sequenceDiagram
    participant Dev as Engineer
    participant CI as CI
    participant Pipe as Pipeline
    participant Score as Scorers
    participant Store as Baseline store
    Dev->>CI: Open PR (prompt/model change)
    CI->>Store: Load golden set + baseline
    loop Each case
        CI->>Pipe: Run system under test
        Pipe-->>CI: Output
        CI->>Score: Deterministic + judge
        Score-->>CI: Per-case metrics
    end
    CI->>Store: Compare aggregates
    alt Regression beyond tolerance
        CI-->>Dev: Fail build + failing cases
    else Within tolerance
        CI-->>Dev: Pass + artifact report
    end

Every PR that can change behavior must re-score the same frozen distribution.

1. Define success criteria — correctness, format, tone, latency; write measurable thresholds.
2. Collect real inputs — 50–200 from logs/tickets covering core and edge paths.
3. Label — domain experts write references/rubrics; engineers automate checks.
4. Implement scorers — deterministic first; add judges for subjective dims; calibrate.
5. Baseline — score current production pipeline; record per-dimension metrics.
6. CI gate — on every PR touching prompts, models, or pipeline; assert thresholds.
7. Expand — every production failure becomes a case; quarterly staleness review.

Real Production Example

CI eval for a customer-support ticket summarizer — format gates plus calibrated faithfulness:

from __future__ import annotations

import json
from dataclasses import dataclass, field
from typing import Callable

from openai import OpenAI

client = OpenAI()


@dataclass
class EvalCase:
    ticket_id: str
    input_text: str
    must_include: list[str] = field(default_factory=list)
    must_not_include: list[str] = field(default_factory=list)
    max_words: int = 150


@dataclass
class EvalResult:
    case_id: str
    passed: bool
    format_score: float
    faithfulness_score: float
    failures: list[str]


class LLMEvaluator:
    def __init__(
        self,
        pipeline_fn: Callable[[str], str],
        *,
        judge_model: str = "gpt-5.6-luna",  # cheap CI judge after calibration
    ):
        self.pipeline = pipeline_fn
        self.judge_model = judge_model

    def score_format(self, output: str, case: EvalCase) -> tuple[float, list[str]]:
        failures: list[str] = []
        words = len(output.split())
        if words > case.max_words:
            failures.append(f"Too long: {words} words (max {case.max_words})")
        for term in case.must_include:
            if term.lower() not in output.lower():
                failures.append(f"Missing required term: {term}")
        for term in case.must_not_include:
            if term.lower() in output.lower():
                failures.append(f"Forbidden term present: {term}")
        denom = max(len(case.must_include) + len(case.must_not_include), 1)
        score = 1.0 - (len(failures) / denom)
        return max(score, 0.0), failures

    def score_faithfulness(self, source: str, output: str) -> float:
        response = client.chat.completions.create(
            model=self.judge_model,
            messages=[{
                "role": "user",
                "content": (
                    "Score faithfulness 0.0-1.0. Is every claim in the summary "
                    "supported by the source ticket? Return only a float.\n\n"
                    f"Source: {source[:3000]}\nSummary: {output}"
                ),
            }],
            temperature=0,
        )
        try:
            return float(response.choices[0].message.content.strip())
        except ValueError:
            return 0.0

    def evaluate_case(self, case: EvalCase) -> EvalResult:
        output = self.pipeline(case.input_text)
        format_score, failures = self.score_format(output, case)
        faithfulness = self.score_faithfulness(case.input_text, output)
        passed = format_score >= 0.8 and faithfulness >= 0.85 and not failures
        return EvalResult(
            case_id=case.ticket_id,
            passed=passed,
            format_score=format_score,
            faithfulness_score=faithfulness,
            failures=failures,
        )

    def run_eval(self, test_set: list[EvalCase], min_pass_rate: float = 0.85) -> dict:
        results = [self.evaluate_case(c) for c in test_set]
        pass_rate = sum(1 for r in results if r.passed) / len(results)
        avg_faith = sum(r.faithfulness_score for r in results) / len(results)
        assert pass_rate >= min_pass_rate, (
            f"Pass rate {pass_rate:.1%} below {min_pass_rate:.1%}"
        )
        return {
            "pass_rate": pass_rate,
            "avg_faithfulness": avg_faith,
            "results": results,
        }


# CI usage
test_set = [EvalCase(**c) for c in json.load(open("golden_test_set.json"))]
evaluator = LLMEvaluator(summarize_ticket)
evaluator.run_eval(test_set, min_pass_rate=0.85)

What this encodes: deterministic gates first, judge for grounding, assert on pass rate so CI cannot be skipped, judge model chosen for cost after human calibration — not a permanent "use GPT-4o" default.

Design Decisions

Common patterns

Decision matrix

When metric drop → what to fix

Comparisons

Common Mistakes

1. Evaluating on demo cases only — three cherry-picked examples prove nothing.
2. Single-metric obsession — 95% format with 60% faithfulness is not success.
3. No baseline comparison — absolute scores without prior version are meaningless.
4. Trusting LLM-as-judge without calibration — measure human agreement first.
5. Static test sets — production drift stale-tests; add failures continuously.
6. Happy-path-only coverage — include adversarial, empty, and out-of-domain inputs.
7. Skipping CI — manual eval gets skipped under deadline pressure.
8. Using leaderboards for product decisions — build domain sets instead.
9. Changing prompt and model together — cannot attribute regressions.
10. Treating judge scores as ground truth — they are proxies; humans remain the calibration source.

Common Mistake

Shipping because "the new model scored higher on a public benchmark" while your golden set was never re-run. Capability ≠ product quality.

Where It Breaks Down

• Creative / subjective tasks — brand voice and tone need human review; judges help but lag taste.
• Non-determinism — temperature > 0 adds variance; use temperature 0 for gates or report confidence intervals across repeats.
• Judge bias — verbosity, position, and self-preference can invert true rankings.
• Expensive labels — start with 50 high-confidence cases; LLM-draft labels only with human verification.
• Metric gaming (Goodhart) — optimizing the suite can diverge from user value; validate with humans and production feedback.
• Multi-turn chat — single-turn cases miss context accumulation errors; add dialogue suites.
• Distribution shift — new product lines invalidate old golden sets until refreshed.

When NOT to Rely on Automated Eval Alone

Do not treat automated CI scores as sufficient when:

• Stakes are material — clinical, legal, financial claims without human review paths
• Labels are contested — experts disagree on what "correct" means; resolve policy first
• Judges are uncalibrated — no human agreement study on your domain
• The task is primarily aesthetic — creative writing, brand voice launches
• You have no golden set — vibes-only shipping is not evaluation

In those cases, either invest in labels + calibration, keep a human gate, or do not automate the quality claim.

Warning

An uncalibrated LLM judge in CI is worse than no gate: it creates false confidence and trains the team to ignore failures.

Running in Production

Best Practice

Publish pass rate and faithfulness on a versioned golden set; block merges on regressions. You cannot manage quality you do not measure.

Production checklist

• Golden set ≥ 50 (prefer 100–200) from real traffic
• Deterministic scorers for format / schema
• Calibrated judge for subjective / faithfulness dims
• Baseline stored with dataset + model + prompt versions
• CI gate on behavior-changing PRs
• Failure cases promoted into the set
• Weekly human sample for calibration
• Clear link to prompt / RAG suites when those surfaces own the failure

Related Guides

Diagram: Evaluation learning path

flowchart LR
    PE[Prompt eng] --> LE[LLM eval]
    LE --> PrE[Prompt eval]
    LE --> RE[RAG eval]
    LE --> HD[Halluc detect]
    LE --> Hub[Evaluation hub]
    RE --> Ret[Retrieval eval]

Start from general LLM eval, then specialize by surface: prompts, RAG, or detection.

Hub and foundations:

• Evaluation — evaluation hub and system-level practice
• Large Language Models — probabilistic generation
• Prompt Engineering — primary lever under test
• Structured Outputs — format compliance paths

Specialize:

• Prompt Evaluation — versioning, A/B, prompt CI
• RAG Evaluation — faithfulness and context metrics
• Agent Evaluation — trajectories and tool-using loops
• Hallucinations — why fluent wrong answers happen
• Hallucination Detection — claim checks and NLI
• Benchmarks — public capability suites (not product gates)
• Observability — online traces and drift

Tools: LangChain · LlamaIndex · ChatGPT · Claude

Interview Questions

1. How do you evaluate LLM outputs in production?
   Golden set + multi-dimensional scorers (deterministic + calibrated judge) + CI regression vs baseline; complement with online sampling.

2. Exact match vs LLM-as-judge?
   Exact/schema for structured fields; judges for open-ended quality. Prefer deterministic when it applies.

3. Pointwise vs pairwise judging?
   Pointwise scores one output vs a rubric; pairwise chooses between two. Pairwise needs order randomization.

4. Why calibrate judges?
   Judges have verbosity/position/self biases. Measure agreement with humans (κ) before CI trust.

5. Golden set size?
   ≥50 for signal; 100–200 for gates; quality and coverage beat raw count.

6. Public benchmarks vs domain eval?
   Benchmarks inform model selection; domain sets decide ship quality for your workload.

7. How do you detect regressions?
   Freeze dataset version; compare aggregates to baseline; fail on threshold breaches; inspect failing cases.

8. Same model as judge and generator?
   Avoid when possible — self-preference bias. Prefer a separate family or tier.

Key Takeaways

• Evaluate the system you ship on a frozen, production-like golden set.
• Mix exact/schema validators with calibrated judges; humans remain the calibration source.
• Separate format, faithfulness, relevance, safety, and ops metrics — one number hides root cause.
• CI gates beat manual spot-checks; expand the set from production failures.
• Leaderboards are not product SLAs.

FAQs

How do you evaluate LLM outputs?

Combine deterministic checks, task validators, LLM-as-judge for subjective dims, and periodic human review on a golden set that mirrors production.

What is LLM-as-judge?

A separate model scores outputs against a rubric (pointwise) or prefers one of two outputs (pairwise). Requires calibration.

How many test cases do I need?

Minimum 50; 100–200 for CI. Coverage of failure modes matters more than raw count.

What faithfulness score is good enough?

Often ≥ 0.90 for factual/RAG paths; ≥ 0.85 during iteration. Tune on your risk tolerance — see RAG evaluation.

Should judge and generator be the same model?

No when avoidable. Self-evaluation biases scores. Use a separate model; cheap judges are fine after calibration.

How do I handle non-determinism?

Temperature 0 for gates, or repeat cases and report intervals. Pin model IDs.

Eval vs monitoring?

Eval = offline fixed set pre-deploy. Monitoring = live metrics and feedback post-deploy. You need both — observability.

How do I build a golden set quickly?

20 real queries labeled by an expert + 10 edges; expand weekly from failures.

Can I fully automate labeling?

Not for high-stakes domains. LLM-assisted drafts with human verification are fine for bootstrap.

How often should I run evals?

Every behavior-changing PR; nightly for drift; full suite before model migrations.

How do I compare two models?

Same golden set, same prompts, pinned settings; compare pass rate, faithfulness, latency, cost on ≥50 cases.

Is human evaluation still necessary?

Yes — for calibration and subjective dimensions. Automation scales; humans keep it honest.

References

• RAGAS Documentation
• DeepEval Documentation
• OpenAI Evals
• LangSmith Evaluation
• Anthropic Documentation

Further Reading

• promptfoo
• Braintrust Eval
• Hugging Face Evaluate