Tau-bench is a benchmark family from Sierra that tests whether a tool-using agent can complete realistic customer service tasks. It matters because it grades the whole conversation, the tool calls, and the policy rules together, which is closer to how production support agents fail.
For example, suppose a simulated airline customer asks your agent to refund a ticket. The agent must look up the booking, check the refund policy, and update the database. Suppose it gets everything right in 6 of 8 trials. Its per-trial success is 75%, but the chance that 4 random trials all succeed is only about 21%.
Tau-bench is also written τ-bench. The name refers to a family of releases, not a single fixed test, so every result depends on the release, user simulator, and metric definition used.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why tau-bench matters for agent evaluation
Most benchmarks grade a single answer. Tau-bench grades a multi-turn interaction in which the agent talks to a user, calls tools, and changes data.
That makes it useful for:
- Testing tool use in context: See whether the agent picks the right tool and arguments while a conversation is still unfolding.
- Checking policy compliance: Confirm that the agent respects domain rules, such as eligibility windows for refunds, along the way.
- Measuring reliability: Find out whether the agent succeeds on the same task every time, not just once.
- Finding failure types: Replay failed conversations to separate misunderstanding, wrong tool use, and policy violations.
The family is known for scoring reliability across repeated trials of the same task. One successful run is weak evidence.
For more on how benchmark results relate to live behavior, see benchmark vs. production evaluation.
How tau-bench works
The original τ-bench was released by Sierra Research. It is described in the tau-bench repository. Each task runs through four steps.
- The agent works in a service domain such as retail or airline support.
- A simulated user, played by a language model, gets a scenario and instructions. It then talks with the agent over multiple turns.
- The agent calls domain tools to look up orders, change a booking, or issue a refund. Those tools read and write an underlying database.
- The task ends when the user or agent closes the conversation.
Grading then compares the final database state with the expected state. It also checks that domain policy rules were respected during the conversation.
This means the answer text alone does not decide the score. An agent can sound helpful and still fail because it changed the wrong record or skipped a policy check.
The simulated user is part of the test
Because the user is simulated, the simulator is part of the measuring instrument. Its model, prompt, and settings affect how cooperative, ambiguous, or persistent the “customer” is.
The same agent can therefore score differently under a different simulator. Treat the simulator as a setting you must record, not as a fixed background.
Later releases change the setup
Tau2-bench adds a dual-control setting. In it, the simulated user can also take actions with tools in a shared environment, rather than only talking to the agent.
Subsequent releases revise domains, tasks, and grading. Scores from different releases must not be averaged or ranked against each other.
How tau-bench measures reliability
The family popularized reporting consistency, not just average success. Suppose the agent attempts a task k independent times. A pass^k-style measure asks how often the agent succeeds on all k trials.
That number falls quickly when an agent is erratic, even if its average success looks respectable. For more on how pass^k differs from pass@k, see pass@k.
This matters in operation. A support agent that resolves refunds correctly in most conversations still mishandles the rest. Those failures create real refunds, real escalations, and real policy breaches.
Worked example: average success vs. consistency
Suppose you run three hypothetical airline tasks 8 times each and record how many trials succeed. The code below applies a count-based pass^k estimator to each task and averages across tasks. It uses only the standard library.
from math import comb
def pass_hat_k(n, c, k):
if c < k:
return 0.0
return comb(c, k) / comb(n, k)
# Hypothetical: trials per task and successful trials.
tasks = {"change_flight": (8, 8), "refund_request": (8, 6), "baggage_claim": (8, 2)}
for k in (1, 2, 4):
scores = [pass_hat_k(n, c, k) for n, c in tasks.values()]
print(f"pass^{k} = {sum(scores) / len(scores):.3f}")
Output:
pass^1 = 0.667
pass^2 = 0.524
pass^4 = 0.405
The agent succeeds on two-thirds of trials. Yet the chance that it succeeds on four trials in a row is about 40%. The baggage task contributes nothing at k = 4, and the refund task drops to about 21%.
When you read any tau-bench result, confirm which metric is reported. Per-trial success, success within k attempts, and success in all k trials give very different numbers from the same run data. The metric definitions belong to each release’s documentation, so cite the release rather than a summary.
How to run and record a tau-bench evaluation
Suppose your team evaluates a support agent on the airline domain. A useful run record pins every setting that can change the score.
benchmark release: <tau-bench or tau2-bench release / commit, pinned>
domains and task set: <version or access date recorded in run config>
user simulator: <model and settings recorded; not assumed from defaults>
trials per task: k, with per-trial outcomes kept, not only the aggregate
grading: final database state + policy checks, per the release definition
follow-up: failed conversations replayed turn by turn from the trace
Treat “tau-bench” in any claim as a family name until these are specified:
- The release.
- The task set version or access date.
- The user simulator.
- The agent model and scaffold.
- The number of trials per task.
- The run date.
Without that list, a quoted result cannot be reproduced or fairly compared.
How to debug tau-bench failures
The follow-up replay is where the benchmark pays off. Read each failed conversation alongside its tool calls. That separates three different defects:
- Misunderstanding: The agent misread what the simulated user wanted.
- Wrong action: The agent chose the wrong tool or passed the wrong arguments.
- Policy violation: The agent broke a rule it could have checked.
Each defect has a different fix. The aggregate score alone cannot tell them apart.
Then look at both ends of the results. Tasks the agent always fails usually point to a capability, tool, or policy-modeling gap. Tasks it fails intermittently usually point to ambiguity handling, state tracking over long conversations, or recovery after a tool error.
Compare candidate agents only when every pinned setting matches. Re-run after any simulator or dataset revision before claiming progress.
That per-run inspection habit is the core of agent-native evaluation.
Limitations of tau-bench
Tau-bench is closer to production than single-answer tests, but it has clear limits.
- Simulated users are tidier than real ones: A simulator cannot reproduce the full range of real customer behavior.
- Overfitting to the simulator: An agent tuned hard against one simulator can learn the simulator rather than the task.
- Limited policy coverage: Grading covers only the policies the benchmark encodes, in the domains it ships. Strong results say nothing about policies unique to your operation.
- Scores are dated: Releases, task sets, and simulators all change, so a score is a measurement from a point in time, not a standing property of a model.
- Conservative agents can be penalized: Repeated-trial grading exposes inconsistency. It can also mark agents that refuse or escalate borderline cases as failing.
Tau-bench vs. single-turn and code benchmarks
Single-turn evaluations grade one answer. Code benchmarks such as SWE-bench grade a patch against tests.
Tau-bench grades a multi-turn interaction. Correctness depends on the final state and on policy compliance along the way. The counterpart in the conversation is itself model-generated.
That makes tau-bench closer to production support work. It also makes results more sensitive to setup, because changing the simulator changes the test. Neither property makes it better or worse in general, but both make pinning the setup necessary.
Frequently asked questions
What is the difference between τ-bench and τ²-bench?
They are different releases in one family, with different task designs and interaction setups. τ²-bench adds dual control, where the simulated user can also act on the shared environment. Results from the two are not interchangeable, so cite the exact release and check its repository for current definitions.
Why does tau-bench run the same task multiple times?
Because a single success can be luck. Repeated trials show whether the agent reliably reaches the correct final state within policy. That is the property a production service needs.
Can I compare a tau-bench score from a vendor page with one from a paper?
Only if the release, task set version, simulator, metric definition, and trial count all match. If any of those is unstated, treat the comparison as unverified.
Does a strong tau-bench result mean an agent is ready for production?
No. It is evidence about behavior in the benchmark’s domains, against a simulated user, under its policies. Validate on your own conversations, tools, and policy rules before drawing production conclusions.
Why did my tau-bench score change when I changed the simulator?
The simulator decides how the customer behaves. A more ambiguous or persistent simulated user creates harder conversations, so the same agent can score differently.
Putting tau-bench to work
Record the full pin list with every run: release, task set, simulator, agent model and scaffold, trials per task, metric definition, and run date. Keep per-trial, per-task outcomes rather than only a headline number.
Pair benchmark reliability with live evaluation metrics and explicit pass/fail criteria on real conversations. Simulated users are systematically tidier than real ones.
Tracing each trial makes the replay step practical. Tools such as Arize AX capture the conversation and tool calls for each run, so you can inspect a failed trial turn by turn. For more on building an evaluation program around these results, see Arize’s LLM evaluation resource and the LLM evaluation glossary entry.
