Terminal-Bench is a benchmark that evaluates AI agents on tasks they complete in a real terminal environment. It matters because it scores whether the agent actually finished the work, not whether its explanation sounds plausible.
For example, suppose two runs use the same model on the same pinned set of 80 tasks. Run A passes 62 tasks (77.5%) and Run B passes 54 (67.5%). Run A’s harness retries a failed command up to three times inside one attempt, while Run B allows one pass. Much of that 10-point gap reflects the more forgiving harness, not only the model.
In a Terminal-Bench task, the agent runs commands, edits files, installs dependencies, and checks its results. Automated checks then inspect the final environment or output to decide whether the task passed.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why Terminal-Bench matters for agent evaluation
Many evaluations grade a single generated answer. Terminal-Bench grades completed work in an environment, which is closer to what an agent does in your own systems.
Terminal-Bench is useful for:
- Testing operational skill: See whether an agent can plan a sequence of commands, recover from failures, and decide when it is actually done.
- Screening agents: Narrow down which model and harness combinations are worth testing on your own tasks.
- Debugging failure types: Use the failed-task list and traces to see whether failures come from tools, file editing, reasoning, or verification.
- Comparing configurations: Measure the effect of a model or harness change while everything else stays fixed.
Arize’s LLM evaluation resources cover the broader evaluation practice that this kind of benchmark fits into.
How Terminal-Bench works
Each task gives the agent a terminal environment and an instruction. The instruction might be to fix a broken build or configure a service.
The agent then acts through shell commands and file changes, often over many steps. Along the way it has to read error output, recover from failed commands, and decide when the task is finished.
The run ends when the agent stops or runs out of time or steps. Automated checks then inspect the result. Depending on the task, they may run tests, compare files, or query process state.
The primary source for the current task set, releases, and rules is Terminal-Bench itself. Check it before citing any result, because task sets and harnesses change between releases.
What must stay fixed when comparing runs
A Terminal-Bench score describes one system under one set of conditions. A comparison between two runs is controlled only if you hold these constant:
- Release and task set: The exact version, and which tasks were included or excluded.
- Harness: The agent framework, loop limits, retry rules, and error handling around the model.
- Model and settings: The model version, temperature or reasoning settings, and context limits.
- Tools and environment: Available commands, network access, installed packages, time limits, and compute.
- Scoring: The checks as shipped for that release, without modification.
If you change any one of these, you are measuring a different system. In the opening example, only the retry rule changed, and the score moved.
A benchmark ranks systems under its own conditions. Whether that ranking holds on your workload is a separate claim that needs its own evidence. For more on that gap, see benchmark vs. production evaluation.
How to report a Terminal-Bench result
A useful report names every condition that affects the score. The following template lists the minimum fields.
benchmark: Terminal-Bench (release pinned in run notes)
task set: full set, no exclusions
harness: <name and version>, max steps and timeout recorded
model: <exact model version>, settings recorded
result: tasks passed / tasks attempted, plus failed-task list
Keep the failed-task list. Ten failures caused by one missing tool are a different problem from ten failures spread across reasoning, file editing, and verification.
The following script groups a run’s failed tasks by cause. The task IDs, causes, and counts are hypothetical.
from collections import Counter
# Hypothetical failed-task list from one run: (task id, failure cause).
failures = [
("task-03", "missing tool"),
("task-11", "missing tool"),
("task-17", "missing tool"),
("task-22", "missing tool"),
("task-29", "file editing"),
("task-34", "missing tool"),
("task-41", "verification"),
("task-48", "missing tool"),
("task-55", "missing tool"),
("task-60", "reasoning"),
]
passed, attempted = 70, 80
print(f"Pass rate: {passed}/{attempted} = {passed / attempted:.1%}")
for cause, count in Counter(cause for _, cause in failures).most_common():
print(f"{cause:<14} {count}")
Output:
Pass rate: 70/80 = 87.5%
missing tool 7
file editing 1
verification 1
reasoning 1
In this run, seven of the ten failures trace back to one missing tool. Fixing the environment would likely move the score more than changing the model.
To assign causes like these, you need to read the traces behind each failure. That is the same work as debugging agents in production.
Worked example: reading two runs side by side
Return to the two hypothetical runs from the opening. Both use the same model and the same 80 tasks.
| Condition | Run A | Run B |
|---|---|---|
| Retries per failed command | Up to 3 within one attempt | None |
| Tasks passed | 62 | 54 |
| Pass rate | 77.5% | 67.5% |
The headline numbers suggest Run A is the better agent. But the harness differs, so the comparison mixes two changes: the model’s behavior and the retry rule.
To isolate the model, rerun both configurations with the same retry rule. To measure the retry rule, keep the model fixed and change only the harness. Either way, report the condition you changed as part of the result.
Terminal-Bench vs. other benchmarks and harnesses
Terminal-Bench is sometimes grouped with coding benchmarks, but the unit of work differs. A code-generation benchmark can score a patch against tests without the agent operating an environment end to end. Terminal-Bench requires those operational steps as well.
It also differs from a general agent benchmark such as AgentBench, which spans several environment types instead of concentrating on terminal work.
An evaluation harness is the machinery that runs a benchmark consistently. Confusing the harness with the benchmark is a common reporting error. Swapping the harness while keeping the task set still changes the result.
Finally, Terminal-Bench is different from evaluating a single model response. It evaluates a multi-step trajectory and its final state. For more on single-response evaluation, see LLM evaluation.
Limitations of Terminal-Bench
A strong benchmark score is not the same as production reliability. An agent can score well on terminal tasks and still fail on yours, because your tools, permissions, data, or definition of done differ.
Other limitations to watch for:
- Contamination and familiarity: Tasks that resemble public material may be easier for reasons unrelated to operational skill.
- Harness gaming: Generous retries, timeouts, or step limits can raise a score without improving the underlying agent.
- Environment sensitivity: Package versions, network access, and timing can change outcomes between runs.
- Narrow success checks: Passing the shipped checks does not prove the work was done safely, cleanly, or in a way a reviewer would accept.
Treat a Terminal-Bench result as a screening signal and a debugging aid, not as proof that an agent is ready for real work.
Frequently asked questions
What kinds of work does Terminal-Bench test?
Terminal-Bench tests work that an agent completes through a terminal. That includes running commands, editing and creating files, installing or configuring software, and verifying the result. Scoring is based on checks against the final state or output, not on the agent’s description of what it did.
Can I compare Terminal-Bench scores from different harnesses or releases?
Not directly. Release, task set, harness, model settings, tools, environment, and scoring all affect the result. A comparison holds up only when those stay fixed, or when you report the difference explicitly as part of the experiment.
Does a high Terminal-Bench score mean an agent is production-ready?
No. A high score shows the agent completed those benchmark tasks under those conditions. Production readiness also depends on your permissions, tools, data, safety requirements, cost, and failure handling. You need to evaluate those on your own tasks.
What should I record with a Terminal-Bench result?
Record the release, task set, harness name and version, model version and settings, tool and network access, and step and time limits. Also record tasks passed, tasks attempted, and the list of failed tasks. Without that context, the headline number cannot be reproduced or fairly compared.
Putting Terminal-Bench to work
Pin the release, task set, harness, model, and environment before you run anything. Report the score together with those conditions and the failed-task list.
Before trusting an agent with real work, run your own tasks that resemble production. Measure cost and failure types, not only pass rate.
Inspect the full trace behind each failure so you can tell a missing tool from a reasoning error. Tools such as Arize Phoenix can capture those multi-step traces so you can review each command the agent ran.
