Codingcodingagentssoftware-engineeringgithub
SWE-Bench
Evaluates systems on real GitHub issues: given a repository and issue, produce a patch that passes the project’s tests.
Last reviewed: 23 July 2026
What it measures
End-to-end software engineering ability — localization, editing, and verification — not isolated code snippets.
Input / Output
Input: repo snapshot + issue description. Output: patch/diff. Metric: % resolved (tests pass).
Evaluation methodology
Agent or model proposes a patch; unit/integration tests from the original PR determine success. Variants: SWE-Bench Verified, Lite, Multimodal.
Metadata
- Task
- Issue resolution / patch generation
- Domain
- Software engineering
- Modality
- Text + code repositories
- Input type
- Repo + GitHub issue
- Output type
- Code patch
- Evaluation type
- Test-suite pass/fail
- Primary metric
- % Resolved
- Secondary metrics
- SWE-Bench Verified, Lite, pass@k
- Paper
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
- GitHub
- swe-bench/SWE-bench
- Dataset
- SWE-Bench
- Leaderboard
- Official live leaderboard (external)
Live leaderboard
Rankings change as new evaluations are published. View current results on the official leaderboard.
View live leaderboard →Datasets
GitHub repositories
Related research
Related guides
Related tools
Related models
Companies
Related rankings
Explore more benchmarks
- HumanEvalCodingClassic function-level Python code generation benchmark: models complete function bodies from docstrings and signatures.
- GAIAAgentsGeneral AI Assistants benchmark: real-world questions requiring tool use, multi-step reasoning, and web/file interaction — hard for both humans and agents.
- WebArenaAgentsRealistic web environment benchmark where agents must complete tasks on self-hosted sites (e.g. e-commerce, CMS, forums) via browser actions.
- LiveBenchGeneralContamination-free LLM benchmark with frequently updated questions across math, coding, reasoning, language, instruction following, and data analysis.
- Arena HardChatChallenging Chatbot Arena prompts packaged for automatic evaluation — a harder preference-style benchmark derived from real user battles.
- CRAGRAGComprehensive RAG benchmark from Meta stressing retrieval failures, outdated knowledge, and answer faithfulness across realistic industry scenarios.