The Brier score measures the accuracy of probability predictions by averaging the squared gap between each predicted probability and the actual outcome. In AI evaluation, it tells you whether the confidence scores from a classifier or LLM judge deserve your trust.
For example, suppose an LLM judge predicts the chance that each of four responses will pass human review: 0.9, 0.8, 0.4, and 0.6. The first two pass and the last two fail. The judge’s Brier score is 0.1425. Always predicting the base rate of 0.5 would score 0.25, so the judge’s probabilities carry real information.
Lower is better. A score of 0 means every prediction was perfect.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Why Brier score matters for AI evaluation
Many AI systems output a probability or confidence, not just a label. You may threshold that confidence, route on it, or let the system abstain when it is unsure. The Brier score tells you whether those numbers are reliable enough to act on.
The Brier score is useful for:
- Validating LLM judges: Check whether a judge’s stated confidence matches how often it is actually right.
- Comparing probabilistic models: Rank models by the overall quality of their probabilities, not just their labels.
- Setting abstain or routing rules: Confirm that confidence is meaningful before you build decisions on it.
- Tracking drift: Watch whether probability quality changes as the input mix changes.
The Brier score is a proper scoring rule. This means it gives the best expected score to predictions that match the true probability. It rewards honest probabilities rather than overconfident or artificially cautious ones.
How the Brier score works
For each case, take the predicted probability, subtract the actual outcome, and square the result. Then average across all cases.
The outcome is coded 1 if the event happened and 0 if it did not. So a prediction of 0.9 on an event that happened has an error of 0.1, and a squared error of 0.01.
Squaring makes confident mistakes costly
Because the penalty is squared, a confident wrong prediction costs far more than a hedged one. A prediction of 0.6 on an event that did not happen costs 0.36. A prediction of 0.4 on the same outcome costs only 0.16.
This does not mean the score rewards caution for its own sake. As a proper scoring rule, it rewards hedging only when the model truly cannot tell cases apart. That is worth knowing when you compare a cautious model with a bold one.
What the score combines
A common misreading is that the Brier score equals calibration. In fact, it combines two qualities.
Calibration means predictions match observed rates. If a model says 70% across many cases, about 70% of those cases should come true.
Discrimination means the model gives different probabilities to cases with different outcomes. A model that predicts the base rate for everything is perfectly calibrated but has no discrimination.
How to calculate the Brier score
The binary form, as defined in Glenn Brier’s 1950 verification work and presented in scikit-learn’s model calibration documentation, is shown below.
Formula:
BS = (1 / N) × Σ (f_i − o_i)² summed over i = 1 to N
Where:
- BS is the Brier score.
- N is the number of forecasts.
- f_i is the predicted probability for case i.
- o_i is the observed outcome for case i, coded 1 if the event happened and 0 if it did not.
For binary forecasts, the score ranges from 0 to 1.
Multi-class form
For problems with more than two classes, the score sums the squared error over every class k for each case.
Formula:
BS = (1 / N) × Σ_i Σ_k (f_ik − o_ik)²
Where:
- f_ik is the predicted probability that case i belongs to class k.
- o_ik is 1 if case i belongs to class k and 0 otherwise.
This form ranges from 0 to 2 rather than 0 to 1. State which form you used whenever you report a score.
Calculate the Brier score in Python
The following code computes the Brier score for the four judge predictions in the opening example. It also computes the base-rate reference, which is the score you get by predicting the overall pass rate for every case.
forecasts = [0.9, 0.8, 0.4, 0.6]
outcomes = [1, 1, 0, 0]
def brier(f, o):
return sum((fi - oi) ** 2 for fi, oi in zip(f, o)) / len(f)
base_rate = sum(outcomes) / len(outcomes)
reference = brier([base_rate] * len(outcomes), outcomes)
print(f"Brier score: {brier(forecasts, outcomes):.4f}")
print(f"Base rate: {base_rate:.2f}")
print(f"Base-rate reference: {reference:.4f}")
Output:
Brier score: 0.1425
Base rate: 0.50
Base-rate reference: 0.2500
The judge scores 0.1425, below the reference of 0.25. Its probabilities are better than ignoring the model and predicting the base rate.
Worked example: scoring four LLM judge predictions
Suppose an LLM judge predicts the probability that each of four responses passes review. Here are the predictions, outcomes, and squared errors:
| Case | Predicted probability (f) | Outcome (o) | Squared error |
|---|---|---|---|
| 1 | 0.9 | 1 | 0.01 |
| 2 | 0.8 | 1 | 0.04 |
| 3 | 0.4 | 0 | 0.16 |
| 4 | 0.6 | 0 | 0.36 |
The total squared error is 0.57. Dividing by 4 cases gives a Brier score of 0.1425.
Cases 3 and 4 account for most of the score. Case 4 alone contributes 0.36, because the judge leaned toward the wrong answer. Case 3 was on the right side of 0.5 but still not confident enough to score well.
Two of the four cases are passes, so the base rate is 0.5. Predicting 0.5 for every case scores 0.25, which is the reference the judge must beat.
How to interpret the Brier score
A Brier score cannot be judged on its own. You need to know what it is up against.
Compare against the base rate
Always compute the base-rate reference on the same data. A model that cannot beat its own base rate adds no probabilistic information.
Decompose the score
The Murphy decomposition splits the binary Brier score into three parts.
Formula:
BS = Reliability − Resolution + Uncertainty
Where:
- Reliability is the calibration term. Lower is better.
- Resolution measures how well forecasts separate cases with different outcome rates. Higher is better, which is why it is subtracted.
- Uncertainty is the score of always predicting the overall base rate. It reflects how hard the task is, not how good the model is.
The decomposition is exact when forecasts take a limited set of discrete values. It is also exact when every forecast in a bin is replaced by its bin value. For continuous forecasts grouped into bins after the fact, the three terms only approximately add up to the score.
Same score, different models
Two models can share a Brier score with different mixes of these parts. One may be well calibrated but timid. The other may be sharper but slightly miscalibrated.
If you care only about calibration, plot predicted versus observed rates by bin and report that too. If you care about overall probability quality, the Brier score is the right single number.
For more on how this fits with other metrics, see evaluation metrics and model performance.
How to measure and debug Brier score in practice
You need recorded probabilities and resolved outcomes on the same cases. For an LLM judge, that means logging the judge’s stated or derived confidence for each case. You also need the eventual human or verified label. This is the workflow behind score models and confidence-aware judging.
Then work through these steps:
- Compute the overall Brier score and the base-rate reference on the same set.
- Bin predictions, for example 0 to 0.1, 0.1 to 0.2, and so on. Compare the mean prediction with the observed frequency in each bin to see where miscalibration lives.
- Slice by segment, such as task type, language, or difficulty. Aggregate calibration can hide opposing errors that cancel out.
- Track the score over time as part of continuous evaluation. Probability quality drifts when the input mix drifts.
Capture confidence at prediction time rather than reconstructing it later. For where this logging belongs, see LLM evaluation and the evaluation pipeline pattern.
Limitations
The Brier score is a useful single number, but it has blind spots.
- Base-rate dependence: Scores are not comparable across datasets with different outcome rates. A 0.12 on one task and a 0.12 on another are different achievements.
- Sample size: You need enough resolved outcomes per segment before bin-level readings mean anything.
- Verbalized confidence: Confidence that an LLM states in words is often poorly calibrated. The Brier score will expose this, but it cannot fix it.
- Timid forecasts: Optimizing the Brier score alone can favor forecasts near the base rate over sharper ones a product would prefer. Pair it with the thresholded decision metric the product actually ships.
Brier score vs. accuracy and log loss
Brier score vs. accuracy
Accuracy throws the probability away. Once you threshold at 0.5, forecasts of 0.51 and 0.99 count the same.
That means accuracy cannot reward better probabilities. Accuracy alone will hide a judge whose confidence is meaningless.
Brier score vs. log loss
Log loss is also a proper scoring rule. But it penalizes confident errors without limit, so a single 0.999 forecast on a wrong outcome can dominate the result.
The Brier score’s penalty is bounded. That makes it gentler and easier to decompose.
Which to choose
Choose the Brier score when you want a bounded, interpretable measure of probability quality. Choose log loss when extreme confident errors must stand out. Use neither when the decision only needs a thresholded label.
Frequently asked questions
Is a Brier score of 0.25 good or bad?
It depends on the base rate. A score of 0.25 is what you get by predicting 0.5 on every binary case. Compare your score with the base-rate reference on the same data. Below it, the forecasts carry information. Above it, they are worse than ignoring the model.
Is the Brier score a calibration metric?
Not purely. It combines calibration (reliability) with discrimination (resolution), measured against the difficulty set by the base rate. Use binned predicted-versus-observed comparisons when calibration alone is the question.
What is the range of the Brier score?
For binary forecasts, the range is 0 to 1, and lower is better. The multi-class form summed over classes ranges from 0 to 2. State which form you report.
When should I use the Brier score for an LLM judge?
Use it when the judge emits a probability or confidence that you plan to threshold, route on, or use to abstain. It tells you whether that confidence deserves trust better than accuracy on thresholded labels does.
Putting the Brier score to work
Log a probability and a resolved outcome for every case you want to score. Report the Brier score together with the base-rate reference, the sample size, and whether you used the binary or multi-class form.
Add a binned calibration plot when you need to see where the model is overconfident or underconfident. Arize AX and Arize Phoenix can store judge outputs and human labels side by side, which gives you the paired data the Brier score needs.
