Codingcodingagentsterminalsoftware-engineering
Terminal-Bench
Terminal-Bench evaluates agents on end-to-end terminal tasks in reproducible container environments. Version 4.1 hardens task verifiers against reward hacking and regrades stored 4.0 artifacts under the updated verifiers.
Last reviewed: 15 November 2026
What it measures
Whether an agent can complete realistic command-line software-engineering and systems tasks, validated by task-specific checkers.
Input / Output
Input: a containerized task environment and natural-language task. Output: terminal actions and resulting filesystem state. Metric: task success under the task verifier.
Evaluation methodology
Agents run inside reproducible task containers. Terminal-Bench 4.1 updates verifiers and regrades prior stored artifacts; it does not introduce a universally comparable replacement score for unrelated benchmark versions or settings.
Metadata
- Task
- Terminal-based software engineering and systems tasks
- Domain
- Coding agents / terminal workflows
- Modality
- Text + shell + container environment
- Input type
- Task prompt + repository/container
- Output type
- Terminal actions and filesystem changes
- Evaluation type
- Task-specific verifier
- Primary metric
- Task success rate
- Secondary metrics
- Version-specific leaderboard score, Confidence interval
- Paper
- Terminal-Bench 4.1
- GitHub
- harbor-framework/terminal-bench
- Dataset
- Terminal-Bench tasks
- Leaderboard
- Official live leaderboard (external)
Live leaderboard
Rankings change as new evaluations are published. View current results on the official leaderboard.
View live leaderboard →Datasets
GitHub repositories
Related research
- Terminal-Bench 4.1
Follow-up
- Terminal-Bench v4.1.0 release
Documentation
Related guides
Related tools
Related models
- GPT-5
OpenAI
- Claude Opus
Anthropic
- Gemini 3.1 Pro
Google
- DeepSeek R1
DeepSeek
- Qwen3
Alibaba
- Llama 4
Meta
Companies
Related rankings
Explore more benchmarks
- SWE-BenchCodingEvaluates systems on real GitHub issues: given a repository and issue, produce a patch that passes the project’s tests.
- GAIAAgentsGeneral AI Assistants benchmark: real-world questions requiring tool use, multi-step reasoning, and web/file interaction — hard for both humans and agents.
- LiveBenchGeneralLiveBench 2.0 evaluates long-horizon AI work across agentic tasks, design, coding, browser use, data analysis, research, and reasoning.
- HumanEvalCodingClassic function-level Python code generation benchmark: models complete function bodies from docstrings and signatures.
- WebArenaAgentsRealistic web environment benchmark where agents must complete tasks on self-hosted sites (e.g. e-commerce, CMS, forums) via browser actions.
- Arena HardChatChallenging Chatbot Arena prompts packaged for automatic evaluation — a harder preference-style benchmark derived from real user battles.