AgentBench is a benchmark suite that measures how well large language models perform as agents, meaning models that take actions in an environment over multiple turns rather than producing a single block of text. Instead of scoring a model on a question and an expected answer, it drops the model into an interactive setting, lets it act, feeds back the result of each action, and checks whether the task was completed. The paper that introduced it is titled “AgentBench: Evaluating LLMs as Agents,” which is also the phrasing most people search for.
The reason it exists is that ordinary language model benchmarks stop being informative the moment a model starts calling tools. A model can write a fluent explanation of how to query a database and still fail to actually query one, because acting requires holding state across turns, reading error messages, recovering from a bad step, and knowing when to stop. AgentBench was one of the earlier attempts to score that capability directly.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- AgentBench evaluates models as agents across several interactive environments, scoring task completion over multiple turns rather than the quality of a single response.
- Each environment defines its own success check, and the headline number is a weighted combination across very different task types, so read it as a rough capability signal rather than a precise measurement.
- The benchmark isolates model capability by holding the scaffolding fixed. Your production agent varies the scaffolding, which is usually where failures actually come from.
- Where an environment’s check is binary, there is no partial credit: one malformed tool call late in a long trajectory can wipe out the episode.
- A benchmark score tells you which model to try first. Tracing and evaluating your own agent on your own tasks tells you whether it works.
How AgentBench works
The suite is a collection of environments rather than a single dataset. Each one gives the model a goal, an action interface, and a loop: the model emits an action, the environment executes it and returns an observation, and the exchange continues until the model finishes, fails, or exhausts a step budget. The environments are grouped into families that stress different skills: system and code grounded settings such as a shell or a database, knowledge grounded settings that require querying structured data, and interactive settings such as games, puzzles, or web tasks. The exact set has changed across releases, so check the composition of the version you are citing.
Two properties make this harder to build than a static benchmark. First, the environment has to be executable and deterministic enough to be reproducible, which rules out most interesting real-world tasks. Second, the model needs a fixed harness: a prompt format, a parser that turns model output into an action, a retry policy, and a step limit. That harness is part of the measurement whether the benchmark says so or not, which is why the same model can post different numbers under different runners. The mechanics of that layer are worth understanding on their own, and they are covered in what an agent harness is and how to evaluate it.
What the score actually measures
Scoring is defined per environment rather than once for the suite. Some environments reduce to a binary check on the final state, others report a graded or progress-style reward, and the headline number is a weighted combination across environments rather than a plain average. Two consequences follow. Where the check is binary, one malformed tool call late in a long trajectory can wipe out the episode. And any single overall figure is a weighting choice, so a model strong on code tasks and weak on web tasks can land next to a model with the opposite profile.
The environments also have to be checkable, so they skew toward tasks with a crisp definition of done, which is a narrower category than the work most production agents are asked to do.
None of that makes the benchmark useless. It makes it a comparison instrument. It tells you that model A handles multi-turn action loops better than model B under identical conditions, which is a genuinely useful thing to know when you are picking a starting model.
Why a benchmark score is not production reliability
The gap between a benchmark result and a working agent is the whole job of AI engineering. A benchmark holds everything constant except the model. In your system, almost nothing is constant: your tools return real errors, your retrieval returns stale documents, your users phrase requests in ways no benchmark author anticipated, and your prompt has been edited by four people. That gap is why teams that ship agents end up building their own AI agent evaluation sets rather than tracking public numbers.
The practical move is to instrument the agent so you can see each step it took, then score those steps against criteria that match your task. That means capturing spans for every model call, tool call, and retrieval, then attaching evaluations to them, which is the workflow described in the developer guide to AI agent tracing and evaluation. The failure modes you find this way are the ones benchmarks were never designed to surface: a tool schema the model consistently misreads, a router that sends a whole category of request down the wrong path, a loop that retries forever on one specific error.
It also matters which shape your agent takes. A single tool-calling loop, a router over specialists, and a long-running planner each fail differently and need different checks.
FAQ
Is AgentBench a dataset, a leaderboard, or a library?
It is closest to a suite of executable environments plus an evaluation protocol. There is code to run models against those environments and reported results for the models the authors tested. Treat published numbers as a snapshot tied to a specific harness and model version rather than a live ranking.
What does AgentBench test that a normal LLM benchmark does not?
Sequential decision making under feedback. A standard benchmark scores one output against one reference. AgentBench scores whether a model can choose an action, read what happened, and adjust. That includes recovering from its own errors, which is a skill that never appears in single-turn scoring.
Does a high AgentBench score mean an agent will work in my product?
No. It means the model is comparatively capable at multi-turn action loops under controlled conditions. Your agent’s reliability depends on your tools, your context, your prompt, and your error handling. Benchmarks constrain those to make comparison fair, which is exactly why the score does not transfer.
What is VisualAgentBench?
It is a related line of work that extends the same idea to multimodal models acting as agents, where observations include images or rendered interfaces rather than text alone. The evaluation logic is the same: give the model an environment, let it act, check the end state.
How should I evaluate my own agent instead?
Start from traces of real sessions, define what a correct trajectory looks like for your top few task types, and score against that. Model-graded evaluation is a common approach once the criteria are written down, and the surrounding practice of watching agents in production is described in what an agent observability platform does.
