Statistical significance means an observed difference is unlikely to be explained by random sampling variation alone. In AI evaluation, it helps you judge whether a new prompt, model, or agent version truly performs differently, or whether the gap could be chance.
For example, suppose you run prompt versions A and B on the same 500 held-out prompts. A passes 390 (78.0%) and B passes 415 (83.0%). A paired test on these results gives a p-value of about 0.003, which is strong evidence that the 5-point gain is real.
A result is usually called significant when its p-value falls below a threshold chosen in advance, most often 0.05.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why statistical significance matters for AI evaluation
Evaluation scores move between runs even when nothing changes. Significance testing helps you tell a real improvement from that ordinary variation.
It is useful for:
- Comparing versions: Check whether a gap between two prompts or models is larger than chance would produce.
- Gating releases: Avoid shipping a change based on a lucky run.
- Planning experiments: Estimate how many cases you need to detect the difference you care about.
- Reviewing slices: Decide whether a gain or loss in one segment is likely to hold.
Significance answers one narrow question. It does not tell you whether a difference is large enough to matter. You need the effect size and its confidence interval for that.
How a p-value works
A p-value is the probability of seeing a difference at least as large as yours, assuming there is no real difference. That assumption is called the null hypothesis.
The National Institute of Standards and Technology (NIST) handbook on hypothesis-testing procedures gives the standard formulation. NIST’s treatment of confidence intervals covers the interval counterpart.
Two misreadings cause most evaluation mistakes.
First, a p-value of 0.03 is not a 97% probability that the new prompt is better. It describes the data under the assumption of no difference. It is not a probability about the difference itself.
Second, “not significant” does not mean “no effect.” It usually means the sample was too small, or the data too noisy, to separate the effect from zero.
Choosing the threshold
The threshold is called alpha, and it is conventionally 0.05. Choose it before you look at results.
Treating 0.049 as a discovery and 0.051 as nothing is a convention, not a rule of nature. The effect size and confidence interval carry more information than which side of the line you landed on.
How to choose the right test
The test must match how you collected the data. Using the wrong one can hide a real effect or invent a false one.
Independent samples
Use an independent-samples test when different cases go to each version, as in live A/B traffic. For pass rates, a two-proportion z-test or a chi-square test is standard.
Formula:
pooled = (k_A + k_B) / (n_A + n_B)
SE = sqrt( pooled × (1 − pooled) × (1/n_A + 1/n_B) )
z = (p̂_B − p̂_A) / SE
Where:
- k_A and k_B are the number of passing cases for versions A and B.
- n_A and n_B are the number of independent cases each version ran on.
- p̂_A and p̂_B are the observed pass rates, k_A / n_A and k_B / n_B.
- pooled is the combined pass rate, assuming no difference between versions.
- SE is the standard error of the difference under that assumption.
- z is the test statistic. You convert it to a p-value with the normal distribution.
For more on running these experiments in production, see A/B testing for LLMs.
Paired samples
Offline evaluation usually runs both versions on the same cases. That pairing removes variation in case difficulty, but only a paired test keeps the benefit.
For pass/fail outcomes, the standard paired test is McNemar’s test. It uses only the discordant cases, where the two versions disagree.
Formula:
b = cases B passes and A fails
c = cases A passes and B fails
Under the null, b ~ Binomial(n = b + c, p = 0.5)
Where:
- b is the count of cases only version B passes.
- c is the count of cases only version A passes.
- n is the total number of discordant cases.
- p = 0.5 means each discordant case is equally likely to favor either version if there is no real difference.
The exact form of the test computes how unlikely a split as uneven as b versus c would be under that binomial model.
For continuous scores such as judge ratings, latency, or cost, use a paired bootstrap or a paired test on the per-case differences. For more on keeping these runs reproducible, see EvalOps.
Clustered cases
Sessions contribute many turns, and users contribute many sessions. A test that treats every turn as independent overstates the sample size and makes results look more significant than they are.
Aggregate to the independent unit first, such as the session or task. Or resample whole units with a cluster bootstrap.
Calculate significance in Python
The following code runs both tests from the examples in this entry using only the standard library. The McNemar function sums binomial probabilities for the smaller discordant count and doubles the result for a two-sided test.
from math import comb, sqrt
from statistics import NormalDist
def mcnemar_exact(b, c):
n = b + c
k = min(b, c)
tail = sum(comb(n, i) for i in range(k + 1)) / 2**n
return min(1.0, 2 * tail)
def two_proportion_z(k_a, n_a, k_b, n_b):
p_a, p_b = k_a / n_a, k_b / n_b
pooled = (k_a + k_b) / (n_a + n_b)
se_null = sqrt(pooled * (1 - pooled) * (1 / n_a + 1 / n_b))
z = (p_b - p_a) / se_null
p_value = 2 * (1 - NormalDist().cdf(abs(z)))
se_diff = sqrt(p_a * (1 - p_a) / n_a + p_b * (1 - p_b) / n_b)
diff = p_b - p_a
return z, p_value, diff - 1.96 * se_diff, diff + 1.96 * se_diff
# Paired design: both versions on the same 500 prompts
b_only, a_only = 45, 20
effect = (b_only - a_only) / 500
print(f"Paired effect: {effect * 100:+.1f} points")
print(f"McNemar exact p = {mcnemar_exact(b_only, a_only):.4f}")
# Independent design: different 200-case samples
z, p, low, high = two_proportion_z(164, 200, 176, 200)
print(f"Independent: z = {z:.2f}, p = {p:.3f}")
print(f"95% CI for B - A: {low * 100:+.1f} to {high * 100:+.1f} points")
Output:
Paired effect: +5.0 points
McNemar exact p = 0.0026
Independent: z = 1.68, p = 0.093
95% CI for B - A: -1.0 to +13.0 points
Worked example: paired vs. independent comparisons
Suppose you run prompt versions A and B on the same 500 held-out prompts. A passes 390 (78.0%) and B passes 415 (83.0%). The headline gain is 5.0 percentage points.
The case-level breakdown looks like this:
| Outcome | Cases |
|---|---|
| Both pass | 370 |
| Only B passes | 45 |
| Only A passes | 20 |
| Both fail | 65 |
The effect is (45 − 20) / 500, which is +5.0 percentage points. McNemar’s exact test on 45 versus 20 discordant cases gives p of about 0.003.
That is strong evidence of a real difference. A 5-point gain is also large enough to matter for most launch decisions.
The same gap without pairing
Now suppose a different experiment. A passes 164 of 200 cases (82%) and B passes 176 of 200 (88%), but on different cases.
The two-proportion test gives z of about 1.68 and p of about 0.093. That is not significant at 0.05. The 95% confidence interval for the difference runs from about −1 to +13 points.
The 6-point gap looks similar to the first example, but it carries much less evidence. The design is unpaired, and each arm has only 200 cases. Sample size and design, not the size of the gap alone, decide what you can conclude.
Report effect, interval, number of cases, and the test used. For more on reporting, see evaluation metrics.
Check the practical side
Before shipping, ask what 5 points means for users. Check what B costs in tokens and latency, and whether the gain holds in key slices.
For keeping that check going after launch, see continuous evaluation.
Common mistakes
Most false discoveries in evaluation come from how a test is run, not from the math.
- Testing many metrics and slices: Test 20 slices at alpha 0.05 and you expect about one false positive even if nothing changed. Choose the primary metric in advance and adjust the rest. The Bonferroni correction divides alpha by the number of tests and is simple but conservative. Holm’s step-down procedure is a less wasteful default.
- Peeking at live tests: Checking a test daily and stopping when p first dips below 0.05 raises the false-positive rate well above 5%. Fix the sample size or duration in advance, or use a sequential-testing method built for continuous monitoring.
- Optional stopping offline: Adding “just 50 more cases” after each borderline result has the same effect as peeking.
- Cherry-picking runs: Large language model (LLM) outputs vary at nonzero temperature. Record generation settings and, where variance matters, treat repeated runs as part of the design. This also shapes offline vs. online evaluation choices.
- Confusing significance with importance: With 200,000 cases, a 0.2-point gain can be highly significant and still not worth B’s extra latency and cost. Set a minimum worthwhile effect before the experiment and judge the interval against it.
Limitations
Every test assumes a sampling model, and agent data often breaks the simple versions.
- Violated assumptions: Clustered turns, retries, shifting traffic, and evaluator noise distort p-values, usually toward false confidence.
- Judge bias: If an LLM judge prefers B’s style, a significant result may reflect the judge’s preference rather than user value.
- Biased samples: Significance cannot fix a test set that does not represent your real traffic.
- Single results: One significant result is weak evidence in a program running hundreds of experiments. Replicate on fresh traffic to separate a lasting gain from a lucky draw.
Statistical significance vs. confidence intervals and effect size
These three answer different questions.
- p-value: Is zero effect consistent with the data?
- Effect size: How large is the difference, in decision units such as pass-rate points, milliseconds, or dollars per task?
- Confidence interval: What range of true effects is consistent with the data?
The confidence interval is what a launch decision needs. Ship when the interval’s lower bound clears your minimum worthwhile effect, not merely when it excludes zero.
An experiment can be significant with an interval of +0.1 to +0.4 points. That effect is real but too small to matter. Another can be non-significant with an interval of −1 to +13 points. That result means the experiment was underpowered, not that there was no gain.
For more on intervals, see confidence interval.
Frequently asked questions
What p-value threshold should I use?
Use 0.05, chosen in advance, for a single primary comparison. For launch decisions with real downside, a better rule is to require the confidence interval to clear a minimum worthwhile effect.
How many cases do I need to detect a 3-point improvement?
It depends on the baseline rate and the design. Paired runs on the same prompts need far fewer cases than independent samples, because case difficulty cancels out.
As a rough guide, detecting a few points near an 80% baseline with independent samples takes thousands of cases per arm. Run a power calculation with your own baseline before you commit.
My result is p = 0.06. Did the change fail?
No, you cannot conclude that. The data are mildly inconsistent with “no effect,” and the study likely lacked power.
Look at the interval. If it runs from slightly negative to clearly worthwhile, collect more data or switch to a paired design.
Should I test at the turn level to get more data?
Only if turns are truly independent, and inside a session they are not. Aggregate to sessions or tasks, or use a cluster bootstrap, and expect a wider interval.
Putting statistical significance to work
Before an experiment, write down the primary metric, the alpha level, the minimum worthwhile effect, and the sample size. Pick a paired design whenever both versions can run on the same cases.
After the experiment, report the effect, its confidence interval, the number and unit of cases, and the test used. Tools such as Arize Phoenix can run the experiments that produce these scores. The Phoenix documentation covers how to set them up.
