LLM Evaluation Guide

TL;DR

• LLM evaluation measures whether model outputs meet your requirements - correctness, format compliance, safety, latency, and cost - not whether the model is "smart" in general.

• Build a golden test set of 50–200 real inputs with expected outputs or rubrics. This is your regression suite for every prompt, model, or pipeline change.

• LLM-as-judge scales subjective evaluation - use a separate model to score faithfulness, relevance, and helpfulness against a rubric. Calibrate against human labels.

• Separate metrics by failure mode - format errors, factual errors, refusal errors, and latency regressions require different tests and thresholds.

• Run evals in CI - block deploys that drop accuracy below threshold.

Manual spot-checking does not survive deadline pressure.

Why This Matters

You shipped a chatbot. Users report it "feels worse" after last week's prompt change. Your PM asks for a number. You have no number.

Without evaluation, every change is a coin flip. Prompt tweaks, model upgrades, temperature adjustments, and context window changes all alter behavior in ways that are invisible until a user complains - or worse, until a compliance issue surfaces. LLM outputs are non-deterministic, subjective, and multi-dimensional. "Does it work?" is not a binary question.

Evaluation turns LLM development from artisan craft into engineering. It gives you:

1. Regression detection - know immediately when a change breaks something
2. Model selection data - compare GPT-4o vs Claude vs Llama on your tasks, not leaderboard scores
3. Quality gates - block deploys that fail eval thresholds
4. Debugging signal - failure cases tell you what to fix (prompt, retrieval, model, post-processing)
5. Stakeholder communication - "accuracy dropped from 91% to 84%" is actionable; "it seems worse" is not

Teams that skip evaluation iterate slower, ship more regressions, and cannot justify model spend. Teams that invest in eval ship faster because they trust their changes.

The Problem LLM Evaluation Solves

LLMs are probabilistic text generators. The same input can produce different outputs. Quality is subjective and task-dependent. There is no compiler to catch errors, no unit test framework built in, and no single "accuracy" metric that covers all failure modes.

Without systematic evaluation:

• You cannot compare models objectively on your use case
• Prompt changes cause silent regressions
• You optimize for demo cases, not production distribution
• You discover failures from user complaints, not before deploy
• You cannot set SLAs or quality targets with engineering rigor

LLM evaluation provides a measurement layer between your application and the model - the same role integration tests play for traditional software.

What Is LLM Evaluation?

LLM evaluation is the practice of measuring model output quality against defined criteria using a combination of:

The core artifact is a golden test set: a curated collection of inputs with expected outputs, rubrics, or reference answers that represent your production task distribution.

# Minimal golden test case
{
    "input": "Summarize this incident report: ...",
    "reference": "Server outage caused by DNS misconfiguration. Duration: 47 minutes.",
    "rubric": {
        "must_include": ["DNS", "47 minutes"],
        "must_not_include": ["customer data breach"],
        "format": "single paragraph, under 100 words"
    }
}

Evaluation runs your pipeline against every test case, scores outputs, aggregates metrics, and compares against baseline.

How LLM Evaluation Works

Evaluation Dimensions

Production LLM systems need multi-dimensional evaluation:

Correctness - Is the answer factually right? For RAG, is it grounded in retrieved context?

Format compliance - Does output match required schema (JSON, markdown, specific fields)?

Completeness - Did it answer all parts of the question?

Safety - Does it refuse harmful requests? Avoid leaking PII?

Consistency - Similar inputs produce similar outputs?

Latency & cost - p50/p95 response time and token usage within budget?

LLM-as-Judge

When human evaluation doesn't scale, use a separate LLM to score outputs against a rubric:

JUDGE_PROMPT = """
You are an evaluation judge. Score the response on faithfulness (0-1):
- 1.0: Every claim is supported by the provided context
- 0.5: Mostly supported, minor unsupported details
- 0.0: Contains claims not in the context

Context: {context}
Question: {question}
Response: {response}

Return JSON: {"faithfulness": 0.0-1.0, "reasoning": "..."}
"""

Critical rules for LLM-as-judge:

1. Use a different model than the one being evaluated (avoid self-bias)
2. Define explicit rubrics with score anchors - "0.5 means X"
3. Calibrate against 50+ human-labeled examples before trusting scores
4. Run each judgment 3 times and take majority vote for high-stakes eval
5. Position bias exists - swap answer order in A/B comparisons

Faithfulness Metrics

Faithfulness measures whether generated text is supported by provided context (critical for RAG):

Faithfulness = (claims entailed by context) / (total claims)

Implementation approaches:

• Claim decomposition - split response into atomic claims, verify each against context via NLI or LLM judge

• RAGAS faithfulness - automated pipeline using LLM to extract statements and verify against context

• Citation verification - require citations, verify cited spans support the claim

Target: faithfulness ≥ 0.90 for production RAG systems handling factual queries.

Golden Test Set Design

Architecture

A production LLM eval system has five components:

Production LLM evaluation runs a versioned dataset through your pipeline, scores outputs with automated metrics, compares against a baseline, and blocks deployment on regressions while logging results to a dashboard.

Source: RAGAS Documentation

Version your test set alongside your code. When you add test cases, bump the dataset version. Track which dataset version produced which scores.

Step-by-Step Flow

Step 1: Define success criteria. What does "good" mean for your use case? Correctness? Format? Tone? Latency under 2s? Write it down as measurable criteria.

Step 2: Collect real inputs. Pull 50–200 representative queries from production logs, support tickets, or manually written cases covering your core use cases and known edge cases.

Step 3: Create labels. For each input, write a reference answer, a rubric, or a pass/fail criterion. Domain experts label; engineers implement automated checks.

Step 4: Implement scorers. Start with deterministic checks (regex, JSON schema validation, keyword presence). Add LLM-as-judge for subjective dimensions. Calibrate judge against human labels.

Step 5: Run baseline eval. Score your current pipeline. This is your benchmark. Document scores per dimension.

Step 6: Integrate into CI. Run eval on every PR touching prompts, models, or pipeline code. Set thresholds: block if accuracy drops > 3% or faithfulness drops > 5%.

Step 7: Expand continuously. Every production failure becomes a new test case. Review test set quarterly for staleness.

Real Production Example

A CI eval pipeline for a customer support summarization system:

import json
import re
from dataclasses import dataclass, field
from openai import OpenAI

client = OpenAI()

@dataclass
class EvalCase:
    ticket_id: str
    input_text: str
    must_include: list[str] = field(default_factory=list)
    must_not_include: list[str] = field(default_factory=list)
    max_words: int = 150

@dataclass
class EvalResult:
    case_id: str
    passed: bool
    format_score: float
    faithfulness_score: float
    failures: list[str]

class LLMEvaluator:
    def __init__(self, pipeline_fn, judge_model: str = "gpt-4o-mini"):
        self.pipeline = pipeline_fn
        self.judge_model = judge_model

    def score_format(self, output: str, case: EvalCase) -> tuple[float, list[str]]:
        failures = []
        word_count = len(output.split())
        if word_count > case.max_words:
            failures.append(f"Too long: {word_count} words (max {case.max_words})")

        for term in case.must_include:
            if term.lower() not in output.lower():
                failures.append(f"Missing required term: {term}")

        for term in case.must_not_include:
            if term.lower() in output.lower():
                failures.append(f"Forbidden term present: {term}")

        score = 1.0 - (len(failures) / max(len(case.must_include) + len(case.must_not_include), 1))
        return max(score, 0.0), failures

    def score_faithfulness(self, source: str, output: str) -> float:
        response = client.chat.completions.create(
            model=self.judge_model,
            messages=[{
                "role": "user",
                "content": f"""Score faithfulness 0.0-1.0. Is every claim in the summary
supported by the source ticket? Return only a float.

Source: {source[:3000]}
Summary: {output}"""
            }],
            temperature=0,
        )
        try:
            return float(response.choices[0].message.content.strip())
        except ValueError:
            return 0.0

    def evaluate_case(self, case: EvalCase) -> EvalResult:
        output = self.pipeline(case.input_text)
        format_score, failures = self.score_format(output, case)
        faithfulness = self.score_faithfulness(case.input_text, output)
        passed = format_score >= 0.8 and faithfulness >= 0.85 and len(failures) == 0

        return EvalResult(
            case_id=case.ticket_id,
            passed=passed,
            format_score=format_score,
            faithfulness_score=faithfulness,
            failures=failures,
        )

    def run_eval(self, test_set: list[EvalCase], min_pass_rate: float = 0.85) -> dict:
        results = [self.evaluate_case(c) for c in test_set]
        pass_rate = sum(1 for r in results if r.passed) / len(results)
        avg_faithfulness = sum(r.faithfulness_score for r in results) / len(results)

        print(f"Pass rate: {pass_rate:.1%} ({sum(1 for r in results if r.passed)}/{len(results)})")
        print(f"Avg faithfulness: {avg_faithfulness:.3f}")

        failures = [r for r in results if not r.passed]
        if failures:
            print(f"\nFailed cases ({len(failures)}):")
            for f in failures[:5]:
                print(f"  {f.case_id}: {f.failures}")

        assert pass_rate >= min_pass_rate, f"Pass rate {pass_rate:.1%} below {min_pass_rate:.1%}"
        return {"pass_rate": pass_rate, "avg_faithfulness": avg_faithfulness, "results": results}

# CI usage
test_set = [EvalCase(**c) for c in json.load(open("golden_test_set.json"))]
evaluator = LLMEvaluator(summarize_ticket)
evaluator.run_eval(test_set, min_pass_rate=0.85)

Design Decisions

⚠ Common Mistakes

1. Evaluating on demo cases only. Three cherry-picked examples prove nothing. Use real production distribution.

2. Single metric obsession. 95% format compliance with 60% faithfulness is not success. Track all dimensions.

3. No baseline comparison. Scores without context are meaningless. Always compare against previous version.

4. Trusting LLM-as-judge without calibration. Run 50 cases through both human and LLM judge. Measure agreement (target κ > 0.7).

5. Static test sets. Production drift makes test sets stale. Add failures continuously; review quarterly.

6. Evaluating only happy paths. Include adversarial inputs, empty inputs, out-of-domain queries, and jailbreak attempts.

7. Skipping CI integration. Manual eval gets skipped. Automate or it won't happen consistently.

8. Using leaderboard benchmarks for product decisions. MMLU scores don't predict your customer support bot quality. Build domain test sets.

Where It Breaks Down

Subjective tasks - Creative writing, tone matching, and brand voice resist automated scoring. Human eval remains necessary for these dimensions.

Non-determinism - Same input, different outputs. Run eval multiple times or use temperature=0 for consistency. Report confidence intervals, not point estimates.

Judge bias - LLM judges favor verbose, confident answers. They exhibit position bias in comparisons. Mitigate with explicit rubrics and order swapping.

Expensive ground truth - Domain expert labeling is slow and costly. Start with 50 high-confidence cases. Use LLM-assisted labeling with human verification.

Metric gaming - Optimizing for eval metrics can degrade real-world quality (Goodhart's law). Periodically validate against human judgment and production feedback.

Multi-turn conversations - Single-turn eval misses context accumulation errors. Build multi-turn test cases for chat applications.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Important

A golden test set is the highest-ROI artifact in LLM engineering. Invest in it before optimizing prompts, switching models, or adding infrastructure.

Ecosystem

• RAGAS: RAG-focused metrics including faithfulness, answer relevance, context precision. Good starting point for RAG pipelines.

• DeepEval: Comprehensive LLM eval framework with 14+ metrics, pytest integration, and CI support.

• LangSmith: Dataset management, eval runs, tracing integration for LangChain apps. Compare runs side-by-side.

• Braintrust: Eval platform with dataset versioning, scoring functions, and regression detection. Strong CI/CD integration.

• promptfoo: CLI tool for prompt A/B testing and red-teaming. YAML-configured eval suites.

• OpenAI Evals: Open-source eval framework with model-graded evals and templates.

Related Technologies

• Prompt Engineering: Prompts are the primary lever eval measures.

• Prompt Evaluation: Focused eval for prompt changes specifically.

• RAG Evaluation: End-to-end RAG metrics including retrieval and generation.

• Hallucination Detection: Specialized faithfulness and grounding checks.

• Benchmarks: General capability benchmarks (MMLU, HumanEval) for model selection.

• Retrieval Evaluation: Evaluate retrieval stage independently.

Learning Path

Prerequisites: Large Language Models · Prompt Engineering

Next topics: RAG Evaluation · Prompt Evaluation · Benchmarks

Estimated time: 50 min · Difficulty: Intermediate

FAQs

How do you evaluate LLM outputs?

Combine deterministic checks (format, schema), task-specific validators (code execution), LLM-as-judge for subjective quality, and periodic human review. Build a golden test set representing your production task distribution.

What is LLM-as-judge?

Using a separate LLM to score another model's output against a rubric. It scales subjective evaluation but requires calibration against human labels. Use a different model than the one being evaluated.

How many test cases do I need?

Minimum 50 for basic confidence. 100–200 for production CI gates. Expand continuously from production failure logs. Quality of cases matters more than quantity.

What faithfulness score is good enough?

≥ 0.90 for factual Q&A and RAG systems. ≥ 0.85 acceptable during iteration. Below 0.80 indicates systematic hallucination - fix before deploying.

Should I use the same model as judge and generator?

No. Self-evaluation introduces bias. Use a separate model - often a cheaper model (gpt-4o-mini) judging a more capable one (gpt-4o) works well.

How do I handle non-deterministic outputs?

Run eval at temperature=0 for consistency. For stochastic pipelines, run each case 3 times and report pass rate across runs. Use confidence intervals on aggregate metrics.

What's the difference between eval and monitoring?

Eval runs offline on a fixed test set before deploy. Monitoring tracks live production metrics (latency, error rate, user feedback). Both are necessary; eval catches regressions pre-deploy, monitoring catches drift post-deploy.

How do I build a golden test set quickly?

Start with 20 real production queries labeled by a domain expert. Add 10 edge cases (empty input, adversarial, out-of-domain). Expand by 5–10 cases per week from production failures.

Can I automate test set creation?

Partially. Use production logs to collect inputs. LLM-assisted labeling with human verification accelerates reference answer creation. Never fully automate labels for high-stakes domains.

How often should I run evals?

On every PR touching prompts, models, or pipeline code. Nightly for monitoring against baseline. Full eval suite before major model migrations.

What metrics should I track for a RAG system?

Retrieval: recall@k. Generation: faithfulness, answer relevance. End-to-end: correctness. Operational: latency p95, cost per query. See RAG Evaluation.

How do I compare two models?

Run the same golden test set through both models with identical prompts. Compare pass rate, faithfulness, latency, and cost. Statistical significance requires 50+ cases - don't conclude from 5 examples.

Is human evaluation still necessary?

Yes. Automated eval scales; human eval calibrates. Weekly review of 20 random cases catches what automation misses - tone, subtle factual errors, UX quality.

How do I integrate eval into CI/CD?

Run eval script as a CI step after unit tests. Fail the build if pass rate drops below threshold. Store scores as artifacts. Compare against main branch baseline.

References

• Hugging Face Transformers Documentation
• LangChain Documentation
• OpenAI API Documentation

Further Reading

• Anthropic Documentation
• Google AI for Developers
• LlamaIndex Documentation
• Pinecone Learning Center

Summary

• LLM evaluation turns probabilistic model behavior into measurable engineering metrics. - Build a golden test set of 50–200 real cases before optimizing anything else. - Use LLM-as-judge for scale, but calibrate against human labels (target κ > 0.7). - Track multiple dimensions: correctness, faithfulness, format, latency, cost. - Run evals in CI on every prompt and model change - automate or it won't happen.

• Leaderboard benchmarks inform model selection; domain test sets determine production quality.