Pass@k is an evaluation metric that estimates the probability that at least one of k attempts at a task succeeds. In agent and code evaluation, it tells you whether a system can solve a task when it gets several tries, which is a different claim from solving it reliably.
For example, suppose a coding agent attempts one task 10 times and 3 attempts pass the tests. Its pass@2 is about 0.53, so a random pair of attempts usually contains a success. Its pass^2 is only about 0.07, so both attempts in a pair rarely pass.
That second number is pass^k. It estimates the probability that all k attempts succeed. Pass@k measures reach. Pass^k measures consistency.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why pass@k matters for agent evaluation
Agents and code models are usually sampled with randomness turned on, so the same task can pass on one run and fail on the next. Pass@k and pass^k turn that variation into two numbers you can report and compare.
They are useful for:
- Measuring capability: Find out whether the system can solve a task at all when it gets several attempts.
- Measuring reliability: Find out whether the system solves the same task every time, which is what a user running it once experiences.
- Comparing systems fairly: Check that two reported scores use the same k before ranking them.
- Planning retry policies: Estimate how much a retry budget helps, and how much it hides.
A high pass@k with a low pass^k means the system can sometimes succeed but does not do so consistently.
For context on where this metric sits in a broader evaluation program, see agent evaluation and evaluation metrics.
How pass@k and pass^k work
Suppose a model attempts the same coding task several times with sampling enabled. A checker runs tests and marks each attempt pass or fail.
Now group the attempts into sets of k. If any attempt in a group passes, that group counts as a pass@k success. If every attempt in the group passes, it counts as a pass^k success.
With one attempt, the two metrics coincide. Pass@1 and pass^1 are both the single-attempt success rate.
As k grows, the two metrics move in opposite directions:
- Pass@k rises as k increases, even if single-attempt quality does not change. More attempts create more chances to succeed once.
- Pass^k never rises as k increases. More attempts create more chances to fail. It stays level only on tasks the system passes every time.
The distinction matters for agents because retries change the claim being made. A benchmark that samples ten attempts and reports that one passed shows capability under repeated sampling. It does not show that a user who runs the agent once will reliably get a correct result.
Anthropic’s Demystifying evals for AI agents discusses this distinction for agent evaluations. Check the formulas below against the checker and sampling design you use.
How to calculate pass@k and pass^k
A common experimental design samples n attempts per task, with n at least as large as k. You then count c, the number of those attempts that passed.
You could pick one group of k attempts and score it. That throws away the other attempts and makes the estimate noisier. The standard unbiased estimator instead asks: if you drew k attempts at random from the n, without replacement, how likely is the group to contain at least one success?
Formula for one task:
pass@k = 1 − C(n − c, k) / C(n, k)
Where:
- n is the number of attempts sampled for the task.
- c is the number of those attempts that passed.
- k is the group size being evaluated.
- C(a, b) is the number of ways to choose b items from a (“a choose b”).
If n − c is less than k, the fraction is treated as zero. Any group of k must then include a success, so pass@k for that task is 1.
Pass^k uses the same idea. It asks how likely a random group of k is to contain only successes.
Formula for one task:
pass^k = C(c, k) / C(n, k)
Where n, c, k, and C(a, b) mean the same as above. If c is less than k, pass^k for that task is zero.
For a benchmark score, compute the per-task value for each task and average across tasks.
Single-task example
Return to the task with n = 10 attempts and c = 3 passes. For k = 2:
pass@2 = 1 − C(7, 2) / C(10, 2) = 1 − 21/45 ≈ 0.53
pass^2 = C(3, 2) / C(10, 2) = 3/45 ≈ 0.07
The system has a reasonable chance that one of two attempts passes. It has a low chance that both pass. Reporting only the first number would overstate reliability.
Calculate pass@k in Python
The following code applies both estimators to a hypothetical set of four tasks, each sampled 10 times. It uses only the standard library.
from math import comb
def pass_at_k(n, c, k):
if n - c < k:
return 1.0
return 1 - comb(n - c, k) / comb(n, k)
def pass_hat_k(n, c, k):
if c < k:
return 0.0
return comb(c, k) / comb(n, k)
# Passing attempts out of n = 10 for each task.
results = {"task_1": 10, "task_2": 7, "task_3": 3, "task_4": 0}
n, k = 10, 3
at_k = [pass_at_k(n, c, k) for c in results.values()]
hat_k = [pass_hat_k(n, c, k) for c in results.values()]
p1 = [c / n for c in results.values()]
print(f"pass@1 = {sum(p1) / len(p1):.3f}")
print(f"pass@{k} = {sum(at_k) / len(at_k):.3f}")
print(f"pass^{k} = {sum(hat_k) / len(hat_k):.3f}")
Output:
pass@1 = 0.500
pass@3 = 0.675
pass^3 = 0.325
The same 40 attempts produce three very different headline numbers. A team that reports only pass@3 would describe this system as passing about two-thirds of tasks. A team that reports pass^3 would say it passes about one-third reliably.
The simplified formulas and their assumption
Teaching material often uses shorter formulas. They assume each attempt is independent and has the same success probability, p.
Formula:
pass@k = 1 − (1 − p)^k
pass^k = p^k
Where:
- p is the probability that a single attempt succeeds.
- k is the number of attempts.
State that assumption when you use these formulas. Real attempts are often not independent. They share the same prompt, model state, tools, environment, and checker. A systematic misunderstanding can make the failures correlated.
The count-based estimator avoids reducing each task to one assumed p. It still depends on the sampled attempts representing the deployment conditions you care about.
Keep the success checker fixed
Both metrics are only as good as the checker that marks each attempt. For code, the checker may be a test suite. For an agent task, it may be checks on the final state plus constraints on side effects.
Change the checker and you change the metric. Keep it fixed across the systems and runs you compare.
For more on defining the checker separately from the sampling metric, see pass/fail criteria and agent run evaluation.
Limitations of pass@k
Pass@k is easy to compute but easy to misread.
- Hidden inconsistency: Users experience individual runs, not the best of a hidden group. If the product runs the agent once, pass@1 and the pattern of failures matter more than pass@10.
- Retry policy mismatch: If the product does retry, measure the deployed retry and selection policy end to end. Do not assume the benchmark grouping matches it.
- No selector included: “One of k passed in a benchmark” is not the same as “the system can identify and return the passing attempt.”
- Checker errors: Tests that miss wrong side effects inflate c. Flaky tests deflate it. Re-run or quarantine flaky tasks, and audit a sample of passes by hand.
- Missing settings: Temperature, model version, tool availability, and task time limits all change the attempts. A pass@k figure without those details cannot be reproduced.
- No cost or latency: A system that needs ten attempts may show a strong pass@10 while being too slow or expensive for the product. Report attempts, token or task cost, and time beside the metric.
- Equal task weighting: Averaging across tasks gives each task equal weight, which may not match production frequency or importance. Report per-category results where risk differs, and show the number of tasks and samples.
Pass@k vs. accuracy
Neither pass@k nor pass^k is accuracy in the usual single-response sense. The exception is pass@1, which equals the observed single-attempt pass rate.
Comparing a pass@10 result from one system with a pass@1 result from another is not a like-for-like comparison. The first gets ten chances per task. The second gets one.
For more on how benchmark numbers relate to what users see, see benchmark vs. production evaluation.
Frequently asked questions
What is the difference between pass@k and pass^k?
Pass@k estimates the chance that at least one of k attempts succeeds. Pass^k estimates the chance that all k attempts succeed. The first measures reach under sampling, and the second measures consistency.
Does pass@1 mean the same as accuracy?
For a pass-or-fail task with one sampled attempt, pass@1 is the fraction of tasks whose attempt passed. That is comparable to accuracy for that checker and task set. It inherits every limitation of the checker and does not describe graded or partial quality.
Why not just run k attempts and count groups directly?
You can, but it wastes samples and gives a noisier estimate. Sampling n attempts per task and applying the combinatorial estimator uses all n results for each task. That gives a more stable estimate for the stated k.
When should I use pass^k instead of pass@k?
Use pass^k when the cost of any failed run is high or when you are making a reliability claim. An agent that must complete the same workflow repeatedly is a typical case. Report it beside pass@1 so readers can see both typical and consistent performance.
Why does my pass@k keep going up as I increase k?
Pass@k always rises or stays level as k grows, because each extra attempt is another chance to succeed. A rising pass@k does not mean the system improved. Compare systems only at the same k.
Putting pass@k to work
When you report pass@k or pass^k, record the task set, the number of samples per task (n), the value of k, the temperature or sampling settings, and the success checker. Without those details, the number cannot be reproduced.
Report pass@1 and pass^k together for any agent you plan to ship. If they differ widely, inspect the failed attempts before trusting the higher number.
Keep per-attempt outcomes, not only the aggregate. Tools such as Arize AX can store each attempt as a trace with its evaluation result, so you can see why the same task passed on one run and failed on the next. For more on running these evaluations continuously, see EvalOps and LLM evaluation.
