LLM Benchmarks - MMLU, HumanEval, MTEB Guide

TL;DR

Benchmarks measure general capability on standardized tasks - they inform model selection but do not predict your production quality.
MMLU tests broad knowledge across 57 subjects. Useful for comparing general-purpose models, not domain-specific performance.
HumanEval tests Python code generation from docstrings. Relevant for coding assistants; pass@k is the standard metric.
MTEB ranks embedding models across 8 task types and 58 datasets. Use it to shortlist embedding models for RAG.
Always validate on your domain test set - leaderboard scores are directional. Your golden test set is the ground truth for production decisions.

Your CTO asks: "Should we switch from GPT-4o to Claude Sonnet?" You open the LMSYS Chatbot Arena leaderboard, see Claude ranked higher, and recommend the switch. Three weeks later, your structured extraction pipeline regresses because Claude handles JSON differently.

Benchmarks are the starting point for model selection, not the endpoint. They compress complex model behavior into single numbers that enable comparison - but those numbers measure performance on tasks that may have nothing to do with your application.

Understanding what each benchmark actually tests - and what it doesn't - prevents costly mis-routing. MMLU measures trivia across 57 academic subjects. It does not measure whether a model follows your system prompt, handles your document format, or stays faithful to retrieved context.

Used correctly, benchmarks narrow a field of 50 models to 3–5 candidates. Your domain eval selects the winner.

The Problem Benchmarks Solve

The LLM landscape has hundreds of models across dozens of providers. Without standardized benchmarks, model comparison is anecdotal - "I tried it and it seemed good" doesn't scale to engineering decisions.

Benchmarks provide:

Standardized comparison - same tasks, same metrics, reproducible results
Capability mapping - which models excel at code, reasoning, knowledge, embeddings
Progress tracking - how fast is the field advancing on specific tasks
Cost-quality tradeoffs - open models approaching closed-model performance

They do not provide:

Performance on your specific use case
Production reliability (latency, uptime, rate limits)
Faithfulness in RAG pipelines
Prompt sensitivity for your task format

What Are LLM Benchmarks?

Benchmarks are standardized test suites with defined tasks, inputs, expected outputs, and scoring metrics. The LLM benchmark ecosystem spans several categories:

Category	Benchmark	What It Measures	Key Metric
Knowledge	MMLU	57-subject multiple choice	Accuracy
Reasoning	GSM8K	Grade-school math word problems	Accuracy
Code	HumanEval	Python function generation	pass@k
Instruction	MT-Bench	Multi-turn instruction following	LLM judge score
Chat	LMSYS Arena	Human preference (Elo rating)	Win rate
Embeddings	MTEB	8 task types, 58 datasets	Mean score
RAG	BEIR	Zero-shot retrieval across 18 datasets	nDCG@10
Long context	Needle in a Haystack	Recall of inserted fact in long doc	Recall
Agents	SWE-bench	Real GitHub issue resolution	Resolve rate

Each benchmark has known limitations - data contamination (models trained on test data), metric gaming, and narrow task scope.

How Benchmarks Work

MMLU (Massive Multitask Language Understanding)

57 subjects (STEM, humanities, social sciences, professional domains). 4-choice multiple choice questions. ~14,000 questions total.

Prompt: "What is the primary function of mitochondria?
(A) Protein synthesis (B) Energy production (C) DNA replication (D) Cell division
Answer:"
Expected: B

Score interpretation:

90%+ - frontier model territory (GPT-4o, Claude Opus)
80–90% - strong general model (GPT-4o-mini, Claude Sonnet)
70–80% - capable open model (Llama 3 70B)
< 70% - limited general knowledge

Limitations: Multiple choice format doesn't reflect production tasks. Models may have seen similar questions in training data. High MMLU doesn't mean good instruction following or RAG faithfulness.

HumanEval

164 Python programming problems. Model receives function signature and docstring, generates function body. Code is executed against unit tests.

pass@k = probability that at least 1 of k generated samples passes all tests

Standard reporting: pass@1 (single attempt) and pass@10 (best of 10).

# HumanEval-style eval
def evaluate_humaneval(problem, generate_fn, k=1):
    passed = 0
    for _ in range(k):
        code = generate_fn(problem["prompt"])
        full_code = problem["prompt"] + code
        if execute_and_test(full_code, problem["test"]):
            passed = 1
            break
    return passed

# pass@1 across all problems
scores = [evaluate_humaneval(p, model.generate, k=1) for p in problems]
pass_at_1 = sum(scores) / len(scores)

Score interpretation:

pass@1 > 90% - frontier coding (o3, Claude Opus)
pass@1 80–90% - strong (GPT-4o, DeepSeek-R1)
pass@1 70–80% - capable (GPT-4o-mini, Codestral)
pass@1 < 60% - not suitable for autonomous code generation

Limitations: Python only, single-function scope. Doesn't test multi-file refactoring, debugging, or codebase understanding. SWE-bench addresses some of these gaps.

MTEB (Massive Text Embedding Benchmark)

Evaluates embedding models across 8 task categories:

Task	Metric	Example
Classification	Accuracy	Sentiment, topic
Clustering	V-measure	Document grouping
Pair Classification	AP	Duplicate detection
Reranking	MAP	Search reranking
Retrieval	nDCG@10	Document search
STS	Spearman correlation	Semantic similarity
Summarization	Spearman correlation	Summary quality
Bitext Mining	F1	Translation matching

Models ranked by mean score across all tasks. Retrieval-specific score most relevant for RAG.

Score interpretation (retrieval):

nDCG@10 > 0.55 - top-tier (text-embedding-3-large, Cohere embed-v3)
nDCG@10 0.45–0.55 - strong (BGE-large, E5-large-v2)
nDCG@10 0.35–0.45 - acceptable for prototyping (all-MiniLM-L6-v2)
nDCG@10 < 0.35 - likely insufficient for production RAG

Critical caveat: MTEB scores are on public datasets, not your corpus. Always run retrieval evaluation on your data before selecting an embedding model.

Other Important Benchmarks

GSM8K - 8,000 grade-school math word problems. Tests chain-of-thought reasoning. Relevant for financial/analytical applications.

MT-Bench - 80 multi-turn questions scored by GPT-4 judge. Tests instruction following and conversation quality.

SWE-bench - Real GitHub issues from popular repos. Model must generate a patch that passes tests. Most realistic coding benchmark.

BEIR - Zero-shot retrieval benchmark across 18 datasets. Predicts embedding model retrieval performance.

RAG couples a dense vector index of external knowledge with a sequence-to-sequence generator. At query time, the retriever selects relevant passages and the generator conditions its answer on that evidence.

Architecture

Using benchmarks in a model selection workflow:

Stage	Input	Output	Time
Leaderboard review	Task requirements	Shortlist of 3–5 models	1 hour
Benchmark deep-dive	Shortlist	Relevant benchmark scores	2 hours
Domain eval	Golden test set	Pass rate, faithfulness per model	1–2 days
Cost analysis	Token pricing, latency	Cost per 1K queries	2 hours
Decision	All above	Selected model + fallback	-

Never skip domain eval. Benchmarks narrow the field; your test set picks the winner.

Step-by-Step Flow

Step 1: Define task requirements. What does your application need? Code generation? Factual Q&A? Structured extraction? Long-context reasoning? Match requirements to benchmark categories.

Step 2: Check leaderboards. LMSYS Arena for chat quality. MTEB for embeddings. HumanEval/SWE-bench for code. MMLU for general knowledge. Shortlist models that score well on relevant benchmarks.

Step 3: Read benchmark limitations. Check for data contamination concerns. Understand what the benchmark doesn't measure (your specific task format, RAG faithfulness, latency).

Step 4: Run domain eval. Test shortlisted models on your golden test set (50–200 cases). Compare pass rate, faithfulness, format compliance, latency, cost.

Step 5: Analyze cost-quality tradeoff. A model 5% better on your eval but 10x more expensive may not be worth it. Plot quality vs cost.

Step 6: Select primary and fallback. Primary model for production. Fallback for rate limits, outages, or cost optimization.

Step 7: Re-evaluate quarterly. New models release monthly. Re-run shortlist eval when major models launch.

Real Production Example

Model selection script comparing candidates on domain eval:

import json
import time
from dataclasses import dataclass
from openai import OpenAI

@dataclass
class ModelCandidate:
    name: str
    provider: str
    model_id: str
    mmlu_score: float | None = None
    cost_per_1m_input: float = 0.0
    cost_per_1m_output: float = 0.0

@dataclass
class ModelEvalResult:
    model: str
    pass_rate: float
    avg_faithfulness: float
    avg_latency_ms: float
    cost_per_1k_queries: float

CANDIDATES = [
    ModelCandidate("GPT-4o", "openai", "gpt-4o", mmlu_score=88.7, cost_per_1m_input=2.50, cost_per_1m_output=10.0),
    ModelCandidate("GPT-4o-mini", "openai", "gpt-4o-mini", mmlu_score=82.0, cost_per_1m_input=0.15, cost_per_1m_output=0.60),
    ModelCandidate("Claude Sonnet", "anthropic", "claude-sonnet-4-20250514", mmlu_score=89.0, cost_per_1m_input=3.0, cost_per_1m_output=15.0),
]

def run_domain_eval(candidate: ModelCandidate, test_set: list[dict], pipeline_fn) -> ModelEvalResult:
    client = OpenAI()  # or Anthropic client for Claude
    passed = 0
    faithfulness_scores = []
    latencies = []
    total_input_tokens = 0
    total_output_tokens = 0

    for case in test_set:
        start = time.time()
        result = pipeline_fn(client, candidate.model_id, case["input"])
        latency = (time.time() - start) * 1000
        latencies.append(latency)

        if case.get("expected_contains"):
            if all(term in result["output"].lower() for term in case["expected_contains"]):
                passed += 1

        if result.get("faithfulness") is not None:
            faithfulness_scores.append(result["faithfulness"])

        total_input_tokens += result.get("input_tokens", 500)
        total_output_tokens += result.get("output_tokens", 200)

    n = len(test_set)
    input_cost = (total_input_tokens / 1_000_000) * candidate.cost_per_1m_input
    output_cost = (total_output_tokens / 1_000_000) * candidate.cost_per_1m_output
    cost_per_1k = (input_cost + output_cost) * (1000 / n)

    return ModelEvalResult(
        model=candidate.name,
        pass_rate=passed / n,
        avg_faithfulness=sum(faithfulness_scores) / len(faithfulness_scores) if faithfulness_scores else 0,
        avg_latency_ms=sum(latencies) / len(latencies),
        cost_per_1k_queries=cost_per_1k,
    )

def compare_models(test_set: list[dict], pipeline_fn):
    results = []
    for candidate in CANDIDATES:
        print(f"Evaluating {candidate.name}...")
        result = run_domain_eval(candidate, test_set, pipeline_fn)
        results.append(result)

    print(f"\n{'Model':<20} {'Pass Rate':>10} {'Faithfulness':>14} {'Latency':>10} {'Cost/1K':>10}")
    print("-" * 70)
    for r in sorted(results, key=lambda x: x.pass_rate, reverse=True):
        print(f"{r.model:<20} {r.pass_rate:>9.1%} {r.avg_faithfulness:>14.3f} {r.avg_latency_ms:>8.0f}ms ${r.cost_per_1k_queries:>8.2f}")

    best = max(results, key=lambda r: r.pass_rate)
    cheapest_acceptable = min(
        (r for r in results if r.pass_rate >= best.pass_rate - 0.03),
        key=lambda r: r.cost_per_1k_queries,
    )
    print(f"\nRecommendation: {cheapest_acceptable.model} (within 3% of best, lowest cost)")

# Usage
test_set = json.load(open("golden_test_set.json"))
compare_models(test_set, my_pipeline)

Design Decisions

Decision	Option A	Option B	When to choose
Benchmark source	Public leaderboards	Self-run benchmarks	Leaderboards for initial screening; self-run for final comparison
Primary benchmark	MMLU (general)	HumanEval (code)	Match to your primary task type
Embedding selection	MTEB retrieval score	Domain retrieval eval	MTEB to shortlist; domain eval to decide
Model count to eval	3 models	10 models	3 for cost efficiency; 10 when requirements are unclear
Eval depth	Benchmark scores only	Full domain eval	Never deploy on benchmark scores alone
Re-eval frequency	Quarterly	On major model releases	Quarterly baseline; immediate eval when frontier models launch

⚠ Common Mistakes

Choosing models based on MMLU alone. High MMLU doesn't mean good instruction following, RAG faithfulness, or JSON output compliance.
Ignoring data contamination. Models trained on benchmark data score artificially high. Prefer benchmarks with held-out test sets and contamination checks.
Not running domain eval. The most expensive mistake. Benchmarks narrow the field; your test set makes the decision.
Comparing total params to active params. Mixtral 8x7B has 47B total but ~13B active. Compare at same active param count for fair benchmarking.
Using MTEB scores for your corpus. Public dataset performance doesn't transfer. Run retrieval eval on your documents.
Optimizing for leaderboard position. Benchmark gaming (training on test data, prompt engineering for benchmark format) doesn't improve production quality.
Ignoring latency and cost. A model 2% better but 5x slower and 10x more expensive rarely wins in production.

Where It Breaks Down

Domain gap - No benchmark tests your specific task. Medical diagnosis, legal contract review, and internal tooling all require custom eval.

Data contamination - Models may have seen benchmark questions during training. Scores are inflated. HELM and live benchmarks mitigate this.

Metric oversimplification - Reducing model capability to one number hides tradeoffs. A model strong at code may be weak at summarization.

Static benchmarks - Benchmarks don't evolve with model capabilities. Models saturate benchmarks (MMLU approaching ceiling), requiring harder benchmarks (MMLU-Pro, GPQA).

Production factors unmeasured - Rate limits, uptime, API latency, context window behavior, and tool-calling reliability aren't captured in any benchmark.

Embedding benchmark transfer - MTEB retrieval scores on Wikipedia and scientific papers don't predict performance on your SaaS documentation corpus.

Running in Production

Best Practice

✅ Best Practices - Instrument every stage, version embedding models, enforce access control at retrieval time, and evaluate on a fixed golden set before shipping changes.

Dimension	Consideration
Scaling	Benchmark eval is offline, one-time or quarterly. Not on query path.
Latency	Running full MMLU (14K questions) takes hours. Use published scores for screening; run domain eval for final selection.
Cost	Domain eval on 200 cases × 3 models: $5–30 depending on models. One-time cost per selection cycle.
Monitoring	Track your domain eval scores over time, not benchmark scores. Alert on domain eval regression after model updates.
Evaluation	Re-run domain eval when switching models, quarterly, or when new frontier models launch.
Security	Domain eval uses your test data - no external benchmark data concerns.

Important

Benchmarks answer "which models are capable?" Your golden test set answers "which model works for us?" Never deploy without the second.

Ecosystem

Hugging Face Open LLM Leaderboard: Automated MMLU, GSM8K, HumanEval evals for open models.
LMSYS Chatbot Arena: Human preference Elo ratings for chat models. Most reliable for conversational quality.
MTEB Leaderboard: Embedding model rankings across 8 task types.
SWE-bench: Real-world coding benchmark with verified GitHub issues.
HELM (Stanford): Comprehensive benchmark suite with transparency about data contamination.
LangSmith / Braintrust: Run domain eval suites against multiple models with comparison dashboards.

LLM Evaluation: Domain-specific eval that complements benchmarks.
Large Language Models: Understanding model architecture informs benchmark interpretation.
Embedding Models: MTEB guides embedding selection; domain eval confirms.
Retrieval Evaluation: Validates embedding model choice on your corpus.
GPT Models: Benchmark scores for OpenAI model family.

Learning Path

Prerequisites: Large Language Models · Embedding Models

Next topics: LLM Evaluation · GPT Models · Embedding Models

Estimated time: 45 min · Difficulty: Intermediate

Benchmarks standardize model comparison but don't predict your production quality. - MMLU for knowledge, HumanEval for code, MTEB for embeddings - use each for its intended purpose. - Shortlist 3–5 models from benchmarks, then run domain eval on your golden test set to decide. - MTEB retrieval scores shortlist embedding models; domain retrieval eval on your corpus confirms.
Ignore benchmark scores that don't match your task type - high MMLU doesn't help a code generation pipeline. - Re-evaluate quarterly as new models launch; the field moves fast.

Next Topics

Learning Path

Continue Learning

Production

Foundations

Retrieval