Prompt Evaluation Guide

TL;DR

• Prompt evaluation treats prompts as code — version them, test against golden sets, and block deploys that regress quality or format.

• Every prompt change can silently break production — a reworded instruction, moved example, or added constraint alters outputs across the entire traffic mix.

• A/B on the same golden set — run baseline vs candidate with the model pinned; compare format compliance, correctness, faithfulness, and refusal behavior separately.

• Template parameters matter — temperature, tool schemas, few-shot slots, and retrieved-context placeholders are part of the prompt contract; eval must include realistic bindings.

• Know when prompt eval is enough — if format/tone/instructions fail, iterate prompts; if knowledge is missing or grounding fails systematically, change retrieval/model/architecture, not just wording. See LLM evaluation and observability.

On this page

• Why This Matters
• The Problem Prompt Evaluation Solves
• How We Got Here
• What Is Prompt Evaluation?
• How Prompt Evaluation Works
• Architecture
• Step-by-Step Flow
• Real Production Example
• Design Decisions
• Comparisons
• Common Mistakes
• Where It Breaks Down
• When NOT to Keep Iterating on Prompts Alone
• Running in Production
• Related Guides
• Interview Questions
• Key Takeaways
• FAQs
• References

Why This Matters

A senior engineer "improves" an extraction prompt by adding "be thorough and comprehensive." Extraction quality drops from 94% to 81%. The model emits extra fields, blows token limits, and breaks downstream parsers. Nobody notices until the data pipeline fails three days later.

Prompts are the highest-leverage and highest-risk component in many LLM apps. A single word change can alter every request. Unlike traditional code, prompt regressions are silent — the model still responds, just differently.

Most teams treat prompts as config: edit a string, deploy, hope. Production teams treat prompts as code: version, test, review, deploy with eval gates. The difference is discovering regressions in CI versus from user complaints. Prompt evaluation is the test framework for your most critical application logic — grounded in prompt engineering craft and LLM evaluation measurement.

The Problem Prompt Evaluation Solves

Prompt work is iterative. You try variants, eyeball outputs, pick what "looks better." That approach:

• Does not scale past a handful of cases
• Is subjective across reviewers
• Misses edge regressions you did not manually check
• Has no baseline — "better" is undefined
• Gets skipped under deadline pressure

Prompt evaluation provides automated regression detection, quantitative A/B, CI integration, version history, and statistical comparison infrastructure.

How We Got Here

Prompts started as notebook cells and Slack pastes. Product scale forced registries, diffs, and gates. The same companies that built LLM evaluation harnesses discovered that most day-to-day quality swings came from prompt edits, not model swaps — so prompt-scoped suites with pinned models became a first-class CI surface.

Diagram: Prompts becoming testable artifacts

timeline
    title From scratchpads to CI-gated prompt versions
    2022 : Inline strings
         : Manual spot checks
    2023 : Prompt hubs / files
         : Ad hoc A/B in notebooks
    2024 : Golden sets per task
         : Format + judge scorers
    2025-2026 : CI on every prompt PR
         : Pin model; promote on gates

Version control arrived before measurement; CI gates arrived once silent regressions became expensive.

What Is Prompt Evaluation?

Prompt evaluation is systematically measuring prompt quality by running versioned prompts on a curated test set and scoring outputs against criteria:

# Prompt eval case
{
    "input": "Extract invoice data from: 'Invoice #4521, Acme Corp, $3,200, due Feb 15'",
    "prompt_version": "extract_v2.3",
    "expected_format": {
        "invoice_id": "str",
        "vendor": "str",
        "amount": "float",
        "due_date": "str"
    },
    "expected_values": {
        "invoice_id": "4521",
        "vendor": "Acme Corp",
        "amount": 3200.0
    },
    "rubric": "Valid JSON only. Never invent fields not present in the input."
}

Engineering Insight

Prompt eval is a specialization of LLM evaluation: the independent variable is the prompt (and its template bindings), with model ID held fixed unless you are deliberately testing a model migration.

How Prompt Evaluation Works

Prompt versioning

Treat prompts as versioned artifacts, not inline strings:

prompts/
├── extract/
│   ├── v2.2.txt          # Current production
│   ├── v2.3.txt          # Candidate
│   └── CHANGELOG.md
├── summarize/
│   ├── v1.0.txt
│   └── v1.1.txt
└── eval/
    ├── extract_golden.json
    └── summarize_golden.json

# prompts/extract/v2.3.txt
SYSTEM: You are a data extraction assistant for Acme Corp.
Extract structured data from invoices. Return JSON only.
If a field is not found in the input, use null — never guess.

SCHEMA: {"invoice_id": str, "vendor": str, "amount": float, "due_date": str}

Version in git. Tag releases. Never edit production prompts in place — create a candidate, eval, then promote.

Template parameters and contracts

A "prompt" in production is usually a template:

• System instructions + user message skeleton
• Few-shot example slots
• Tool / function schemas (function calling)
• Retrieved context placeholders (RAG)
• Output schema hints (structured outputs)
• Decoding params (temperature, max tokens) — pin in eval metadata

Eval cases must bind these realistically. Testing an extraction prompt without messy inputs, or a RAG prompt without weak/strong contexts, underestimates failure modes.

Treat each template variable as part of the contract under test:

If two environments bind templates differently (staging truncates context; prod does not), prompt eval on staging will lie. Mirror production binding rules in CI.

Scoring dimensions

Track separately — a prompt can improve one while breaking another:

A/B prompt comparison

def compare_prompts(prompt_a, prompt_b, test_set, scorer):
    results_a = [scorer(prompt_a, case) for case in test_set]
    results_b = [scorer(prompt_b, case) for case in test_set]
    score_a = sum(r.passed for r in results_a) / len(results_a)
    score_b = sum(r.passed for r in results_b) / len(results_b)
    regressions = [
        {"input": case["input"], "a": ra.output, "b": rb.output}
        for case, ra, rb in zip(test_set, results_a, results_b)
        if ra.passed and not rb.passed
    ]
    return {
        "score_a": score_a,
        "score_b": score_b,
        "delta": score_b - score_a,
        "regressions": regressions,
    }

Promote B only if score improves or stays within tolerance and regressions are reviewed (some tradeoffs are acceptable; format breaks rarely are).

Diagram: Prompt change lifecycle

stateDiagram-v2
    [*] --> Draft: new version file
    Draft --> Eval: CI golden A/B
    Eval --> Rejected: regression / format fail
    Eval --> Review: within tolerance
    Review --> Rejected: unacceptable case fails
    Review --> Promoted: approve + tag
    Rejected --> Draft: revise candidate
    Promoted --> Prod: traffic uses new version
    Prod --> Draft: next change

Candidates never overwrite prod until eval and review pass.

Architecture

Diagram: CI prompt evaluation

flowchart TB
    PR[Prompt PR] --> Diff[Diff vN vs vN+1]
    Diff --> Load[Load golden set]
    Load --> RunA[Run baseline prompt]
    Load --> RunB[Run candidate prompt]
    RunA --> Score[Scorers: format + quality]
    RunB --> Score
    Score --> Cmp{Delta vs tolerance}
    Cmp -->|fail| Block[Block merge + list cases]
    Cmp -->|pass| Approve[Approve + update baseline]

Pin model ID in the job so the diff attributes to the prompt, not a silent provider update.

Source: promptfoo

Step-by-Step Flow

Diagram: A/B eval sequence

sequenceDiagram
    participant Dev as Engineer
    participant CI as CI
    participant LLM as Pinned model
    participant S as Scorers
    Dev->>CI: PR candidate prompt vN+1
    CI->>LLM: Baseline vN × golden set
    CI->>LLM: Candidate vN+1 × same set
    LLM-->>CI: Outputs A/B
    CI->>S: Format + correctness + rubric
    S-->>CI: Pass rates + regressions
    alt Regressions exceed policy
        CI-->>Dev: Fail with case diffs
    else Within policy
        CI-->>Dev: Pass — ready to promote
    end

Identical inputs and pinned model make the prompt the only intentional variable.

1. Version prompts — move off inline strings into files/registry.
2. Golden set per prompt/task — 30–100+ cases; happy, edge, adversarial, refusal.
3. Scorers — deterministic first; judges for subjective dims (LLM evaluation).
4. Baseline — score production prompt; record per-dimension metrics.
5. Eval every change — candidate version; A/B; block if dimension drops beyond tolerance (e.g., 3%) or format fails.
6. Review regressions case-by-case — aggregates hide painful single failures.
7. Promote — merge, tag, update baseline; never silent overwrite.

Real Production Example

CI pipeline with A/B comparison for structured extraction:

from __future__ import annotations

import json
import re
from dataclasses import dataclass, field
from pathlib import Path

from openai import OpenAI

client = OpenAI()


@dataclass
class PromptEvalCase:
    input: str
    expected_fields: dict = field(default_factory=dict)
    must_include: list[str] = field(default_factory=list)
    must_not_include: list[str] = field(default_factory=list)
    output_format: str = "json"


@dataclass
class PromptEvalResult:
    case_input: str
    passed: bool
    format_valid: bool
    correctness_score: float
    output: str
    failures: list[str]


class PromptEvaluator:
    def __init__(self, model: str = "gpt-5.6-luna"):
        self.model = model  # pinned for prompt-only diffs

    def load_prompt(self, path: str) -> str:
        return Path(path).read_text()

    def run_prompt(self, system_prompt: str, user_input: str) -> str:
        resp = client.chat.completions.create(
            model=self.model,
            messages=[
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": user_input},
            ],
            temperature=0,
        )
        return resp.choices[0].message.content or ""

    def score_format(self, output: str, fmt: str) -> bool:
        if fmt != "json":
            return True
        try:
            json.loads(output)
            return True
        except json.JSONDecodeError:
            match = re.search(r"\{.*\}", output, re.DOTALL)
            if not match:
                return False
            try:
                json.loads(match.group())
                return True
            except json.JSONDecodeError:
                return False

    def score_correctness(
        self, output: str, case: PromptEvalCase
    ) -> tuple[float, list[str]]:
        failures: list[str] = []
        try:
            parsed = json.loads(output)
        except json.JSONDecodeError:
            match = re.search(r"\{.*\}", output, re.DOTALL)
            parsed = json.loads(match.group()) if match else {}

        correct = 0
        total = len(case.expected_fields)
        for field_name, expected in case.expected_fields.items():
            actual = parsed.get(field_name)
            if str(actual) == str(expected):
                correct += 1
            else:
                failures.append(
                    f"Field '{field_name}': expected {expected}, got {actual}"
                )

        for term in case.must_include:
            if term.lower() not in output.lower():
                failures.append(f"Missing required: {term}")
                total += 1
            else:
                correct += 1
                total += 1

        for term in case.must_not_include:
            if term.lower() in output.lower():
                failures.append(f"Forbidden term found: {term}")

        score = correct / total if total > 0 else 0.0
        return score, failures

    def evaluate_prompt(
        self, prompt_path: str, test_set: list[PromptEvalCase]
    ) -> list[PromptEvalResult]:
        system_prompt = self.load_prompt(prompt_path)
        results: list[PromptEvalResult] = []
        for case in test_set:
            output = self.run_prompt(system_prompt, case.input)
            format_valid = self.score_format(output, case.output_format)
            correctness, failures = self.score_correctness(output, case)
            if not format_valid:
                failures.append("Invalid output format")
            passed = format_valid and correctness >= 0.8 and not failures
            results.append(
                PromptEvalResult(
                    case_input=case.input,
                    passed=passed,
                    format_valid=format_valid,
                    correctness_score=correctness,
                    output=output,
                    failures=failures,
                )
            )
        return results

    def compare_versions(
        self,
        baseline_path: str,
        candidate_path: str,
        test_set: list[PromptEvalCase],
        max_regression: float = 0.03,
    ) -> dict:
        baseline_results = self.evaluate_prompt(baseline_path, test_set)
        candidate_results = self.evaluate_prompt(candidate_path, test_set)
        baseline_pass = sum(r.passed for r in baseline_results) / len(baseline_results)
        candidate_pass = sum(r.passed for r in candidate_results) / len(candidate_results)
        regressions = []
        for case, br, cr in zip(test_set, baseline_results, candidate_results):
            if br.passed and not cr.passed:
                regressions.append({
                    "input": case.input[:100],
                    "failures": cr.failures,
                })
        delta = candidate_pass - baseline_pass
        approved = delta >= -max_regression and len(regressions) <= len(test_set) * 0.05
        assert approved, (
            f"Prompt regression: {candidate_pass:.1%} vs baseline {baseline_pass:.1%}"
        )
        return {
            "baseline_score": baseline_pass,
            "candidate_score": candidate_pass,
            "delta": delta,
            "regressions": regressions,
            "approved": approved,
        }


test_set = [
    PromptEvalCase(**c)
    for c in json.load(open("prompts/eval/extract_golden.json"))
]
PromptEvaluator(model="gpt-5.6-luna").compare_versions(
    baseline_path="prompts/extract/v2.2.txt",
    candidate_path="prompts/extract/v2.3.txt",
    test_set=test_set,
    max_regression=0.03,
)

For subjective tone suites, add a calibrated judge (Claude Sonnet 5 or GPT-5.6 Sol offline; Haiku 4.5 / Gemini 3.7 Flash-class in CI after κ check) — never a permanent unlabeled "GPT-4o default."

Design Decisions

Common patterns

Decision matrix

When prompt eval is enough vs need model / RAG changes

Comparisons

Common Mistakes

1. Editing prompts in place — no revert, no proof.
2. No golden set — three examples are not a suite.
3. Scoring without baseline — 88% means nothing alone.
4. Changing prompt and model together — attribution dies.
5. Happy-path-only cases — empty, malformed, adversarial, out-of-scope.
6. Ignoring format regressions — better prose that breaks JSON is a prod outage.
7. Floating model in prompt CI — provider updates masquerade as prompt wins/losses.
8. Skipping eval under time pressure — automate so it cannot be skipped.
9. Endless prompt fiddling for knowledge gaps — escalate to RAG/model.

Common Mistake

Merging a prompt because "two examples looked better" while format compliance on the golden set dropped 12%. Downstream parsers are part of the product.

Where It Breaks Down

• Subjective quality — tone/brand need humans; judges require calibration.
• Model drift — scores move when the underlying model changes; re-baseline.
• Test set staleness — refresh from production quarterly+.
• Interaction effects — system + user + tools + retrieved context interact; include realistic bindings.
• Non-determinism — temperature > 0; use 0 for gates or multi-run means.
• Cost — 100 cases × 2 versions = 200 calls per PR; budget intentionally.
• Multi-turn — need dialogue suites, not only single turns.

When NOT to Keep Iterating on Prompts Alone

Stop prompt-only iteration when:

• Evidence is missing — retrieval/context empty or wrong (RAG evaluation)
• Task exceeds model tier — consistent reasoning failures on pinned strong prompts
• Safety/policy needs runtime enforcement — guardrails, not instructions alone
• Structured consumers need hard schemas — prefer constrained decoding / structured outputs
• You lack a golden set — more wording without measurement is noise

Warning

If faithfulness is low because context recall is low, another week of prompt synonyms will not fix it.

Running in Production

Best Practice

Pin the model during prompt eval. Prompt eval measures prompt changes — re-baseline after model migrations.

Production checklist

• Prompts versioned (no silent in-place edits)
• Per-task golden set ≥ 30 (prefer 50–100)
• Format scorers hard-gated
• A/B vs baseline on every prompt PR
• Model ID pinned in CI metadata
• Regression policy documented (0% vs 3%)
• Escalation path when prompt eval plateaus
• Prod failures flow back into the set
• Template bindings in CI match production truncation / tool injection
• Model-migration checklist re-runs all task suites before cutting traffic

Prompt eval vs live traffic A/B

Offline golden A/B is the merge gate. Live traffic A/B (1–5% canary) validates distribution shift the golden set missed — new languages, longer tickets, adversarial users. Require the offline gate first; never canary a prompt that failed format compliance on the suite. Pair canaries with observability dashboards: error rate, parse-fail rate, escalation rate, and token percentiles. Rollback must be a version pin, not a scramble to remember yesterday's wording.

Related Guides

Diagram: Prompt evaluation learning path

flowchart LR
    PE[Prompt eng] --> PrE[Prompt eval]
    PrE --> LE[LLM eval]
    LE --> Hub[Evaluation hub]
    PrE --> Obs[Observability]
    PrE --> SO[Structured outputs]
    PrE --> RE[RAG eval]

Craft prompts, measure them, share scorers with LLM eval, and escalate to RAG/model when wording is not the bottleneck.

Core:

• Prompt Engineering — craft under test
• LLM Evaluation — judges, golden sets, CI patterns
• Evaluation — hub
• Observability — online traces and drift

Adjacent:

• Structured Outputs — format contracts
• Function Calling — tool-prompt scoring
• RAG Evaluation — when context is the variable

Tools: LangChain · ChatGPT · Claude

Interview Questions

1. How do you evaluate prompts?
   Version them; golden set; deterministic + judge scorers; A/B vs baseline; CI gate.

2. How many cases per prompt?
   ≥30 basic; 50–100 for production gates; cover edges and refusals.

3. Why pin the model?
   So score deltas attribute to the prompt, not provider/model drift.

4. Regression tolerance?
   Often 0% for format/extraction; ~3% for open-ended — never ignore format breaks.

5. Prompt eval vs LLM eval?
   Prompt eval isolates prompt versions; LLM eval is the broader measurement toolkit.

6. When is prompting not enough?
   Missing knowledge (RAG/tools), capability ceiling (model), or hard safety (guardrails).

7. How do you eval RAG prompts?
   Bind real contexts; score faithfulness + abstention on insufficient evidence.

8. Git files vs registry?
   Git for eng-owned; registry when non-engineers must edit without code deploys.

Key Takeaways

• Prompts are versioned code with golden-set tests and CI gates.
• A/B with a pinned model is the core comparison pattern.
• Track format and quality separately; format regressions ship outages.
• Escalate when metrics show retrieval or model limits, not wording limits.
• Feed production failures back into the suite continuously.

FAQs

How do you evaluate prompts?

Version, golden-set score, A/B vs baseline, CI gate on regressions.

How many test cases?

Minimum ~30; 50–100 for gates.

Eval prompts separately from models?

Yes for attribution. Pin one variable at a time.

Good regression tolerance?

0% for structured paths; small band for open-ended; never for broken schemas.

Subjective quality?

Calibrated LLM-as-judge + human samples.

Without a golden set?

Not reliably — build at least ~30 cases first.

RAG prompts?

Include contexts; score faithfulness and refusal.

Tools?

promptfoo, Braintrust, LangSmith, or custom pytest — start simple.

Multi-turn?

Dialogue golden sets; score turns and whole conversation.

When to re-baseline?

After every model migration; quarterly otherwise; when prod failures show gaps.

Latency impact?

Track output tokens; verbosity can break cost/latency SLOs even if "quality" rises.

References

• promptfoo Documentation
• OpenAI Prompt Engineering Guide
• Anthropic Prompt Engineering
• LangSmith Prompt Hub
• OpenAI Evals

Further Reading

• Braintrust Evals
• DeepEval
• Google Gemini Prompting