What Is a Confidence Interval?

A confidence interval (CI) is a range of values that shows the uncertainty around a metric calculated from sample data. In AI evaluation, confidence intervals help you determine how much weight to give a reported pass rate, accuracy score, or performance improvement.

For example, suppose an AI agent passes 164 out of 200 evaluation tasks. Its observed pass rate is 82%, but that does not mean it will succeed on exactly 82% of future tasks. A 95% confidence interval of 76.1% to 86.7% describes the sampling uncertainty around that estimate, assuming the tasks were sampled independently and represent the workload you want to measure.

Try Arize AX

Build better agents with Arize

Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.

Get started View AX docs

Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.

Why confidence intervals matter for AI evaluation

An evaluation score tells you what happened on your test cases. A confidence interval helps you understand how precisely those cases estimate performance on a broader set of tasks.

This becomes important when comparing prompts, models, or agent configurations. A change from 82% to 85% accuracy might indicate an improvement, but it could also reflect ordinary sampling variation. You need to understand the uncertainty before drawing a conclusion.

Confidence intervals are useful for:

• Evaluating model and agent performance: Understand how precisely a test set estimates the underlying success or error rate.
• Comparing versions: Determine the range of improvement or regression consistent with your evaluation results.
• Making release decisions: Check whether the evidence supports a minimum quality threshold.
• Planning evaluation datasets: Estimate whether you need more independent test cases to make a reliable comparison.

They are especially useful when your evaluation dataset is relatively small or when performance differences between versions are narrow.

How to interpret a 95% confidence interval

Suppose you evaluate an agent on 200 independently sampled tasks and observe the following result:

Pass rate: 82% (95% CI: 76.1%–86.7%, n = 200)

The 82% is your observed pass rate. The interval indicates that underlying pass rates between approximately 76% and 87% are consistent with the data under the statistical model.

The 95% confidence level describes the reliability of the interval-building procedure. If you repeatedly sampled tasks from the same population and computed an interval each time, approximately 95% of those intervals would contain the true pass rate.

It does not assign a 95% probability to the true value falling inside this particular calculated interval.

For an engineering decision, the distinction is practical. If your release requirement is an underlying pass rate of at least 85%, this evaluation does not establish that you have reached it. The interval includes values below your threshold.

How to calculate a confidence interval

The right calculation depends on what you are measuring. Pass/fail metrics use different methods from continuous metrics such as latency or cost.

Confidence interval for a pass rate or accuracy score

For binary metrics such as task success, accuracy, or error rate, start with the proportion of cases that pass.

Formula (plain text):

p̂ = k / n

Where:

• p̂ is the observed proportion.
• k is the number of passing cases.
• n is the number of independent evaluated cases.

For a 95% confidence interval, the Wilson score method is a useful default. It performs better than the basic normal approximation when sample sizes are small or observed rates are close to 0% or 100%.

Wilson score interval (plain text):

Wilson CI =
  [p̂ + z²/(2n) ± z√(p̂(1−p̂)/n + z²/(4n²))]
  / [1 + z²/n]

For a 95% confidence interval, z is approximately 1.96. The ± symbol produces the lower and upper bounds.

Calculate a Wilson confidence interval in Python

The following Python function calculates a Wilson interval using only the standard library.

Language: Python 3

from math import sqrt
from statistics import NormalDist


def wilson_interval(passed, total, confidence=0.95):
    if total < 1 or not 0 <= passed <= total:
        raise ValueError("Invalid pass count or total")

    z = NormalDist().inv_cdf((1 + confidence) / 2)
    p = passed / total

    denominator = 1 + z**2 / total
    center = (p + z**2 / (2 * total)) / denominator

    margin = (z / denominator) * sqrt(
        p * (1 - p) / total
        + z**2 / (4 * total**2)
    )

    return center - margin, center + margin


passed = 164
total = 200

low, high = wilson_interval(passed, total)

print(
    f"{passed / total:.1%} "
    f"(95% CI {low:.1%} to {high:.1%}, n={total})"
)

Output:

82.0% (95% CI 76.1% to 86.7%, n=200)

The result is more useful than reporting 82% alone because it makes the uncertainty visible.

Confidence interval for continuous metrics

For metrics such as average latency, cost per run, or numerical evaluation scores, a conventional confidence interval for the mean uses the sample standard deviation and a critical value from the t-distribution.

Formula (plain text):

95% CI = x̄ ± t* × (s / √n)

Where:

• x̄ is the sample mean.
• s is the sample standard deviation.
• n is the number of independent observations.
• t* is the appropriate critical value from the t-distribution.

This method assumes independent observations and an approximately normal sampling distribution of the mean. For highly skewed latency or cost distributions, consider a bootstrap interval and report percentile metrics such as p95 when they better describe performance.

Comparing two model or prompt versions

Confidence intervals become especially valuable when deciding whether a new version of an AI system performs better than the previous one.

Suppose you evaluate two agent versions on the same 200 tasks:

Version B has an observed improvement of 6 percentage points.

However, the two pass rates alone do not tell you how certain that improvement is.

Because both versions ran on the same tasks, you should calculate a confidence interval for the paired difference. This preserves the relationship between each task’s result in A and its result in B.

For this example, suppose:

• 156 tasks pass in both versions.
• 20 tasks fail in A but pass in B.
• 8 tasks pass in A but fail in B.
• 16 tasks fail in both versions.

Version B improves 20 cases while regressing on 8, producing a net gain of 12 successful tasks.

Using a paired percentile bootstrap, the estimated 95% confidence interval for the improvement is approximately +1 to +11 percentage points.

This suggests a positive improvement under the sampling assumptions, although the actual benefit could be substantially smaller than the observed 6-point gain.

For more on comparing AI system versions, see A/B testing for LLMs.

Calculate a paired confidence interval in Python

A paired bootstrap repeatedly resamples the evaluation tasks while keeping each task’s results from both versions together.

Language: Python 3 (NumPy)

import numpy as np

# Each pair represents [A passed, B passed].
pairs = np.array(
    [[1, 1]] * 156
    + [[1, 0]] * 8
    + [[0, 1]] * 20
    + [[0, 0]] * 16
)

rng = np.random.default_rng(42)

# Resample paired tasks 10,000 times.
indices = rng.integers(
    0,
    len(pairs),
    size=(10_000, len(pairs))
)

# Calculate B minus A for every resample.
differences = (
    pairs[indices, 1] - pairs[indices, 0]
).mean(axis=1)

low, high = np.percentile(
    differences,
    [2.5, 97.5]
)

observed = (
    pairs[:, 1] - pairs[:, 0]
).mean()

print(
    f"B - A: {observed * 100:+.1f} percentage points"
)

print(
    f"95% paired bootstrap CI: "
    f"{low * 100:+.1f} to {high * 100:+.1f} pp"
)

Output:

B - A: +6.0 percentage points
95% paired bootstrap CI: +1.0 to +11.0 pp

This is an illustrative percentile bootstrap for independent paired tasks. For very small datasets or sparse binary outcomes, methods specifically designed for paired proportions may provide more reliable coverage.

Accounting for sessions, retries, and correlated results

One of the easiest ways to produce misleadingly narrow confidence intervals is to count observations that are not independent.

Consider an agent evaluation containing 2,000 turns across 80 user sessions.

Those 2,000 turns do not necessarily provide the same amount of independent information as 2,000 separate sessions. Turns within one session share context, user behavior, and possibly the same underlying failure conditions.

If you treat every turn as an independent observation, your confidence interval may substantially understate uncertainty.

A cluster bootstrap helps account for this dependence.

Instead of independently resampling individual turns, resample complete sessions with replacement. Keep the turns associated with each sampled session together, then recalculate the metric for every resampled dataset.

For example, if you have 80 independent sessions, each bootstrap sample contains 80 sessions drawn with replacement from the original set.

Repeat this process thousands of times and use the resulting metric distribution to estimate your interval.

Choose your resampling unit carefully. If multiple sessions belong to the same user and share important dependencies, you may need to resample users instead. Also distinguish session-level success rates from turn-weighted success rates because they measure different outcomes.

What confidence intervals cannot tell you

A confidence interval describes sampling uncertainty under a set of statistical assumptions. Several important sources of evaluation error remain outside its scope.

A biased evaluation dataset: If your test set excludes difficult production tasks, a narrow confidence interval can still describe an overly optimistic estimate. Confidence intervals do not correct unrepresentative data.

An unreliable evaluator: An LLM judge that systematically marks incorrect answers as correct can produce a precise estimate of the wrong metric. Evaluator calibration and validation still matter.

Production distribution shift: Performance on one task distribution may not generalize to new users, different inputs, or changed tools and retrieval data.

Run-to-run variability: Stochastic models may produce different results when the same task is repeated. If that variability matters, design your evaluation to measure it rather than assuming one run per task captures it.

These are reasons to combine confidence intervals with representative evaluation datasets, reliable evaluation metrics, and repeatable experiments.

Confidence intervals vs. statistical significance

A statistical significance test evaluates evidence against a specified null hypothesis. A confidence interval helps communicate the size and uncertainty of an estimated effect.

Suppose a change improves an agent’s success rate by 0.5 percentage points and the estimated confidence interval excludes zero. The result may be statistically significant, but that alone does not establish whether the improvement justifies the additional cost, latency, or complexity.

For AI engineering decisions, consider the observed improvement, its confidence interval, and the minimum improvement that would actually matter.

When showing error bars in charts, label them explicitly. Standard deviation, standard error, and confidence intervals measure different quantities and should not be used interchangeably.

Frequently asked questions

How many evaluation cases do I need for a useful confidence interval?

It depends on the precision you need and the kind of metric you are measuring.

For a pass rate, confidence interval width generally decreases as the number of independent cases increases. As a rough rule, cutting the margin of error in half requires approximately four times as many independent cases.

If your interval is too wide to determine whether an agent meets its launch threshold, you may need more evaluation data.

Can I compare two models by checking whether their confidence intervals overlap?

You should calculate a confidence interval for the difference between the models.

Overlapping confidence intervals do not necessarily mean the models perform equally well. When both models run on the same evaluation cases, a paired analysis can provide a more precise comparison.

What confidence level should I use?

A 95% confidence level is a common default for reporting evaluation results.

A 99% interval is generally wider, while a 90% interval is narrower. Choose the confidence level before reviewing your results, and report it consistently.

Can I calculate a confidence interval on a fixed evaluation dataset?

Yes, but the interpretation requires care.

Your observed score is exact for the cases and runs you evaluated. A confidence interval for performance on a broader workload requires assumptions about how those cases represent the target population.

If your dataset is a hand-picked collection of edge cases or regression tests, a confidence interval calculated from those rows alone cannot establish how the system will perform across production traffic.

Why did my confidence interval get wider after accounting for sessions?

Because correlated observations often contain less independent information than their row count suggests.

When multiple turns share the same session, treating them as separate independent cases can produce intervals that are too narrow. Accounting for the session structure gives a more defensible uncertainty estimate.

Putting confidence intervals to work

A good evaluation report should include the observed metric, its confidence interval, the number and unit of evaluated cases, and the method used to calculate the interval.

When comparing versions, report the confidence interval for the difference alongside the raw scores.

Tools such as Arize AX and Arize Phoenix can help you run and inspect evaluations, while statistical analysis of those results helps determine whether an observed change is supported by the data.

The goal is to make decisions based on measurable performance differences while being explicit about what the evaluation data can support.