Prompt Evaluation Guide

TL;DR

• Prompt evaluation treats prompts as code - version them, test them against golden sets, and block deploys that regress quality.

• Every prompt change can silently break production behavior - a reworded instruction, moved example, or added constraint alters outputs in unpredictable ways.

• A/B test prompts on the same golden test set - run old and new prompts against identical inputs, compare scores side-by-side.

• Track format compliance, correctness, and faithfulness separately - a prompt that improves tone but breaks JSON output is a regression.

• Integrate into CI - prompt files are code files. Eval runs on every prompt change, just like unit tests.

Why This Matters

Your senior engineer "improves" the extraction prompt by adding "be thorough and comprehensive." Extraction quality drops from 94% to 81%. The model now includes irrelevant fields, exceeds token limits, and breaks downstream parsing. Nobody noticed until the data pipeline failed three days later.

Prompts are the highest-leverage and highest-risk component in LLM applications. A single word change can alter behavior across every request. Unlike traditional code where a syntax error crashes immediately, prompt regressions are silent - the model still responds, just differently.

Most teams treat prompts as configuration: edit a string, deploy, hope. Production teams treat prompts as code: version, test, review, deploy with eval gates. The difference is whether you discover regressions in CI or from user complaints.

Prompt evaluation is the testing framework for your most critical application logic.

The Problem Prompt Evaluation Solves

Prompt engineering is iterative. You try variations, compare outputs manually, pick what "looks better." This approach:

• Doesn't scale beyond 5–10 test cases
• Is subjective - different reviewers prefer different outputs
• Misses regressions on edge cases you didn't manually check
• Has no baseline - you can't quantify "better"
• Gets skipped under deadline pressure

Prompt evaluation provides:

1. Automated regression detection - every prompt change scored against a golden test set
2. Quantitative comparison - "prompt v2.3 scores 91% vs v2.2 at 94%" is actionable
3. CI integration - block deploys that drop below threshold
4. Version history - track which prompt version produced which scores
5. A/B testing infrastructure - compare prompt variants with statistical rigor

What Is Prompt Evaluation?

Prompt evaluation is the systematic process of measuring prompt quality by running prompts against a curated test set and scoring outputs against defined criteria. It encompasses:

# Prompt eval case
{
    "input": "Extract invoice data from: 'Invoice #4521, Acme Corp, $3,200, due Feb 15'",
    "prompt_version": "extract_v2.3",
    "expected_format": {"invoice_id": "str", "vendor": "str", "amount": "float", "due_date": "str"},
    "expected_values": {"invoice_id": "4521", "vendor": "Acme Corp", "amount": 3200.0},
    "rubric": "Must return valid JSON. Must not include fields not in the input."
}

How Prompt Evaluation Works

Prompt Versioning

Treat prompts as versioned artifacts, not inline strings:

prompts/
├── extract/
│   ├── v2.2.txt          # Current production
│   ├── v2.3.txt          # Candidate
│   └── CHANGELOG.md
├── summarize/
│   ├── v1.0.txt
│   └── v1.1.txt
└── eval/
    ├── extract_golden.json
    └── summarize_golden.json

# prompts/extract/v2.3.txt
SYSTEM: You are a data extraction assistant for Acme Corp.
Extract structured data from invoices. Return JSON only.
If a field is not found in the input, use null - never guess or infer.

SCHEMA: {"invoice_id": str, "vendor": str, "amount": float, "due_date": str}

Version in git. Tag releases. Never edit production prompts in place - create a new version, eval, then promote.

Scoring Dimensions

Evaluate prompts across multiple dimensions simultaneously:

Format compliance - Does output match required structure? JSON schema validation, regex patterns, field presence checks.

Correctness - Are extracted/generated values accurate? Exact match, fuzzy match, or LLM-as-judge.

Completeness - Did the prompt produce all required fields/content?

Consistency - Same input at temperature=0 produces same output across runs?

Faithfulness - For RAG prompts, is output grounded in provided context?

Conciseness - Does output stay within length limits?

Refusal behavior - Does the prompt correctly refuse out-of-scope requests?

A/B Prompt Comparison

Run two prompt versions against the same test set:

def compare_prompts(prompt_a, prompt_b, test_set, scorer):
    results_a = [scorer(prompt_a, case) for case in test_set]
    results_b = [scorer(prompt_b, case) for case in test_set]

    score_a = sum(r.passed for r in results_a) / len(results_a)
    score_b = sum(r.passed for r in results_b) / len(results_b)

    regressions = []
    for case, ra, rb in zip(test_set, results_a, results_b):
        if ra.passed and not rb.passed:
            regressions.append({"input": case["input"], "version_a": ra.output, "version_b": rb.output})

    return {
        "score_a": score_a,
        "score_b": score_b,
        "delta": score_b - score_a,
        "regressions": regressions,
        "improvements": [c for c, ra, rb in zip(test_set, results_a, results_b) if not ra.passed and rb.passed],
    }

Only promote prompt B if score improves or stays within tolerance AND regressions are acceptable.

Architecture

A prompt evaluation system has four components:

Prompt evaluation in CI runs baseline and candidate prompt versions against a golden test set on every pull request, comparing scores and blocking merges when quality regresses.

Source: promptfoo

Step-by-Step Flow

Step 1: Version your prompts. Move prompts from inline strings to versioned files in git. One file per prompt version per task.

Step 2: Build a golden test set per prompt. 30–100 inputs with expected outputs, format requirements, or rubrics. Cover happy path, edge cases, and adversarial inputs.

Step 3: Implement scorers. Start with deterministic checks (JSON schema, regex, field presence). Add LLM-as-judge for subjective dimensions.

Step 4: Establish baseline. Run current production prompt against test set. Record scores per dimension. This is your regression threshold.

Step 5: Eval on every change. When modifying a prompt, create a new version, run eval, compare against baseline. Block if any dimension drops > 3%.

Step 6: Review regressions individually. Aggregate score hides per-case failures. Read every regression case - some are acceptable tradeoffs, others are blockers.

Step 7: Promote with confidence. Merge prompt changes that pass eval gates. Tag the version. Update baseline scores.

Real Production Example

CI pipeline for prompt evaluation with A/B comparison:

import json
import re
from dataclasses import dataclass, field
from pathlib import Path
from openai import OpenAI

client = OpenAI()

@dataclass
class PromptEvalCase:
    input: str
    expected_fields: dict = field(default_factory=dict)
    must_include: list[str] = field(default_factory=list)
    must_not_include: list[str] = field(default_factory=list)
    output_format: str = "json"

@dataclass
class PromptEvalResult:
    case_input: str
    passed: bool
    format_valid: bool
    correctness_score: float
    output: str
    failures: list[str]

class PromptEvaluator:
    def __init__(self, model: str = "gpt-4o-mini"):
        self.model = model

    def load_prompt(self, path: str) -> str:
        return Path(path).read_text()

    def run_prompt(self, system_prompt: str, user_input: str) -> str:
        resp = client.chat.completions.create(
            model=self.model,
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": user_input},
            ],
            temperature=0,
        )
        return resp.choices[0].message.content

def score_format(self, output: str, fmt: str) -> bool:
        if fmt == "json":
            try:
                json.loads(output)
                return True
            except json.JSONDecodeError:
                match = re.search(r'\{.*\}', output, re.DOTALL)
                if match:
                    try:
                        json.loads(match.group())
                        return True
                    except json.JSONDecodeError:
                        pass
                return False
        return True

    def score_correctness(self, output: str, case: PromptEvalCase) -> tuple[float, list[str]]:
        failures = []
        try:
            parsed = json.loads(output)
        except json.JSONDecodeError:
            match = re.search(r'\{.*\}', output, re.DOTALL)
            parsed = json.loads(match.group()) if match else {}

        correct = 0
        total = len(case.expected_fields)
        for field_name, expected in case.expected_fields.items():
            actual = parsed.get(field_name)
            if str(actual) == str(expected):
                correct += 1
            else:
                failures.append(f"Field '{field_name}': expected {expected}, got {actual}")

        for term in case.must_include:
            if term.lower() not in output.lower():
                failures.append(f"Missing required: {term}")
                total += 1
            else:
                correct += 1
                total += 1

        for term in case.must_not_include:
            if term.lower() in output.lower():
                failures.append(f"Forbidden term found: {term}")

        score = correct / total if total > 0 else 0.0
        return score, failures

    def evaluate_prompt(self, prompt_path: str, test_set: list[PromptEvalCase]) -> list[PromptEvalResult]:
        system_prompt = self.load_prompt(prompt_path)
        results = []

        for case in test_set:
            output = self.run_prompt(system_prompt, case.input)
            format_valid = self.score_format(output, case.output_format)
            correctness, failures = self.score_correctness(output, case)

            if not format_valid:
                failures.append("Invalid output format")

            passed = format_valid and correctness >= 0.8 and len(failures) == 0
            results.append(PromptEvalResult(
                case_input=case.input,
                passed=passed,
                format_valid=format_valid,
                correctness_score=correctness,
                output=output,
                failures=failures,
            ))

        return results

    def compare_versions(
        self,
        baseline_path: str,
        candidate_path: str,
        test_set: list[PromptEvalCase],
        max_regression: float = 0.03,
    ) -> dict:
        baseline_results = self.evaluate_prompt(baseline_path, test_set)
        candidate_results = self.evaluate_prompt(candidate_path, test_set)

        baseline_pass = sum(1 for r in baseline_results if r.passed) / len(baseline_results)
        candidate_pass = sum(1 for r in candidate_results if r.passed) / len(candidate_results)

        regressions = []
        for case, br, cr in zip(test_set, baseline_results, candidate_results):
            if br.passed and not cr.passed:
                regressions.append({
                    "input": case.input[:100],
                    "baseline_output": br.output[:200],
                    "candidate_output": cr.output[:200],
                    "failures": cr.failures,
                })

        delta = candidate_pass - baseline_pass
        approved = delta >= -max_regression and len(regressions) <= len(test_set) * 0.05

        print(f"Baseline ({baseline_path}): {baseline_pass:.1%}")
        print(f"Candidate ({candidate_path}): {candidate_pass:.1%}")
        print(f"Delta: {delta:+.1%}")
        print(f"Regressions: {len(regressions)}")
        print(f"Decision: {'APPROVE' if approved else 'REJECT'}")

        if regressions:
            print("\nRegression details:")
            for r in regressions[:5]:
                print(f"  Input: {r['input']}")
                print(f"  Failures: {r['failures']}")

        assert approved, f"Prompt regression: {candidate_pass:.1%} vs baseline {baseline_pass:.1%}"
        return {
            "baseline_score": baseline_pass,
            "candidate_score": candidate_pass,
            "delta": delta,
            "regressions": regressions,
            "approved": approved,
        }

# CI usage
test_set = [PromptEvalCase(**c) for c in json.load(open("prompts/eval/extract_golden.json"))]
evaluator = PromptEvaluator()
evaluator.compare_versions(
    baseline_path="prompts/extract/v2.2.txt",
    candidate_path="prompts/extract/v2.3.txt",
    test_set=test_set,
    max_regression=0.03,
)

Design Decisions

⚠ Common Mistakes

1. Editing prompts in place. Changing the production prompt without versioning. When it breaks, you can't revert. Always create a new version.

2. No golden test set. Evaluating by manually checking 3 examples. Minimum 30 cases per prompt, 50+ for production gates.

3. Evaluating without baseline comparison. Scoring a prompt at 88% means nothing without knowing the previous version scored 94%.

4. Changing prompt and model simultaneously. Can't attribute regression to prompt or model. Change one variable at a time.

5. Only checking happy paths. Edge cases cause production failures: empty input, malformed input, adversarial input, out-of-scope requests.

6. Ignoring format regressions. A prompt that improves content quality but breaks JSON output is a net regression if downstream parsing fails.

7. Not pinning model in eval. Model updates change behavior independently of prompts. Pin model version in eval runs for consistent comparison.

8. Skipping eval under time pressure. The moment you skip eval is the moment you ship a regression. Automate in CI so it can't be skipped.

Where It Breaks Down

Subjective quality - Tone, style, and creativity resist automated scoring. LLM-as-judge helps but requires calibration. Human review remains necessary for subjective dimensions.

Model drift - Prompt eval scores change when the underlying model updates, even without prompt changes. Re-establish baselines after model migrations.

Test set staleness - Production inputs evolve. A test set built six months ago may not represent current input distribution. Refresh quarterly.

Interaction effects - System prompt + user message + retrieved context interact non-linearly. Evaluating prompts in isolation misses context-dependent behavior. Include realistic context in test cases.

Non-determinism - Temperature > 0 introduces variance. Run eval at temperature=0 for consistency. For stochastic prompts, run 3 times and report mean pass rate.

Cost at scale - Evaluating 100 cases × 2 prompt versions × LLM calls = 200 API calls per PR. Budget $1–5 per eval run. Acceptable for quality gates.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Important

Pin the model version during prompt eval. Prompt eval measures prompt changes, not model changes. Re-baseline after model migrations.

Ecosystem

• promptfoo: CLI/YAML prompt evaluation with A/B comparison, red-teaming, and CI integration. Strongest open-source option for prompt eval.

• Braintrust: Eval platform with prompt versioning, scoring functions, and regression detection. Good for team workflows.

• LangSmith: Prompt hub with eval runs linked to prompt versions. Integrated with LangChain.

• DeepEval: Pytest-based eval with prompt comparison support.

• OpenAI Evals: Template-based eval framework with model-graded scoring.

• Weights & Biases Prompts: Prompt versioning and comparison with experiment tracking.

Related Technologies

• Prompt Engineering: The craft that prompt evaluation measures.

• LLM Evaluation: General eval framework; prompt eval is a specialized subset.

• Function Calling: Prompt eval for tool-use prompts requires execution-based scoring.

• Structured Outputs: Format compliance is a key prompt eval dimension.

• Observability: Production monitoring complements pre-deploy prompt eval.

Learning Path

Prerequisites: Prompt Engineering · LLM Evaluation

Next topics: LLM Evaluation · Function Calling · Structured Outputs

Estimated time: 45 min · Difficulty: Intermediate

FAQs

How do you evaluate prompts?

Version prompts in git. Build a golden test set per prompt. Run automated scorers (format, correctness, faithfulness). A/B compare new versions against production baseline. Block deploys that regress.

How many test cases per prompt?

Minimum 30 for basic coverage. 50–100 for production CI gates. Include happy path, edge cases, adversarial inputs, and format boundary cases.

Should I eval prompts separately from models?

Yes. Pin the model version during prompt eval. When evaluating a model change, pin the prompt. Change one variable at a time to attribute regressions correctly.

What is a good regression tolerance?

0% for structured extraction and format-critical prompts. 3% for open-ended generation where minor quality tradeoffs are acceptable. Never accept format compliance regressions.

How do I eval subjective prompt quality?

Use LLM-as-judge with explicit rubrics. Calibrate against human labels on 20+ cases. Track tone, helpfulness, and clarity as separate dimensions alongside objective metrics.

Can I eval prompts without a golden test set?

Not reliably. Manual spot-checking of 3–5 examples misses regressions. Build at least 30 cases before deploying prompt changes to production.

How do I handle prompt eval for RAG prompts?

Include retrieved context in test cases. Evaluate faithfulness (grounded in context) alongside correctness. Test cases where context is insufficient - prompt should produce "I don't know."

What tools should I use for prompt eval?

promptfoo for CLI-based A/B testing. Braintrust for team workflows with dataset management. Custom pytest scripts for full control. Start with promptfoo or custom scripts.

How do I eval multi-turn conversation prompts?

Build multi-turn test cases with conversation history. Evaluate each turn independently and the conversation holistically. Track context retention across turns.

How often should I re-baseline prompt eval scores?

After every model migration. Quarterly for monitoring. Immediately when production failure patterns suggest test set gaps.

Should prompts live in code or a registry?

Git files for engineering teams (version control, PR review, CI integration). Prompt registry/database when non-engineers need to edit prompts without code deploys. Many teams start with git and migrate to registry later.

How do I eval prompt changes that affect latency?

Track token count of outputs alongside quality scores. A prompt that improves quality but doubles output length may exceed context or cost budgets. Include max_tokens compliance in scoring.

References

• Hugging Face Transformers Documentation
• LangChain Documentation
• OpenAI API Documentation

Further Reading

• Anthropic Documentation
• Google AI for Developers
• LlamaIndex Documentation
• Pinecone Learning Center

Summary

• Prompts are code - version, test, review, and deploy with the same rigor as application logic. - A/B compare every prompt change against the production baseline on a golden test set. - Track format compliance, correctness, and faithfulness as separate dimensions - a prompt can improve one while regressing another. - Pin model version during prompt eval to isolate prompt changes from model changes.

• Automate eval in CI on every prompt PR - manual eval gets skipped under deadline pressure. - Every production prompt failure becomes a new test case - expand the golden set continuously.