Codingcodingcode-generationpythonclassic
HumanEval
Classic function-level Python code generation benchmark: models complete function bodies from docstrings and signatures.
Last reviewed: 23 July 2026
What it measures
Syntactic and functional correctness of short Python programs under unit tests (pass@k).
Input / Output
Input: function signature + docstring. Output: function body. Metric: pass@k against hidden tests.
Evaluation methodology
Execute generated code in a sandbox against unit tests. Report pass@1 / pass@k. Often paired with MBPP and LiveCodeBench for broader coding eval.
Metadata
- Task
- Function-level code generation
- Domain
- Python programming
- Modality
- Text / code
- Input type
- Signature + docstring
- Output type
- Function body
- Evaluation type
- Unit-test execution
- Primary metric
- pass@1
- Secondary metrics
- pass@10, pass@100
- Paper
- Evaluating Large Language Models Trained on Code
- GitHub
- openai/human-eval
- Dataset
- OpenAI HumanEval
- Leaderboard
- Official live leaderboard (external)
Live leaderboard
Rankings change as new evaluations are published. View current results on the official leaderboard.
View live leaderboard →Datasets
GitHub repositories
Related research
Related guides
Related tools
Related models
Companies
Related rankings
Explore more benchmarks
- SWE-BenchCodingEvaluates systems on real GitHub issues: given a repository and issue, produce a patch that passes the project’s tests.
- LiveBenchGeneralContamination-free LLM benchmark with frequently updated questions across math, coding, reasoning, language, instruction following, and data analysis.
- Arena HardChatChallenging Chatbot Arena prompts packaged for automatic evaluation — a harder preference-style benchmark derived from real user battles.
- CRAGRAGComprehensive RAG benchmark from Meta stressing retrieval failures, outdated knowledge, and answer faithfulness across realistic industry scenarios.
- GAIAAgentsGeneral AI Assistants benchmark: real-world questions requiring tool use, multi-step reasoning, and web/file interaction — hard for both humans and agents.
- GPQAScienceGraduate-Level Google-Proof Q&A: PhD-level science questions designed so that non-experts with web search still struggle, testing deep domain reasoning.