Mean square error (MSE) is the average of the squared differences between what a model predicted and what actually happened. Take each residual, square it, add them up, divide by the number of observations. Lower is better, and zero means the model reproduced every target exactly.
MSE = (1/n) * sum( (y_i - y_hat_i)^2 ) for i = 1 to nHere y_i is the true value for observation i, y_hat_i is the prediction, and n is the number of observations. MSE is the default loss for regression models, and it is usually the exact quantity gradient descent is minimizing when the target is continuous.
The squaring is not cosmetic. It decides the whole character of the metric. It forces every error to be positive so overshoots and undershoots cannot cancel, it punishes large misses far more than small ones, and it leaves the answer in squared units of the target. That last property is why most teams report the square root instead.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- MSE is the mean of squared residuals. It is never negative, and it equals zero only for a perfect fit.
- Squaring makes large errors dominate: one prediction off by 10 contributes as much as 100 predictions off by 1.
- MSE is measured in squared units of the target, so an MSE of 4 on a dollar model is 4 squared dollars, which is not a number you can explain to a stakeholder.
- Use MSE when large errors really are disproportionately worse. Use mean absolute error when they are not, because MSE will chase outliers and label noise.
- MSE pools across groups with a count-weighted average. RMSE does not, which matters when you roll up per-segment performance into one number.
How to calculate MSE
Suppose a model predicts three delivery times in hours and you later observe the truth.
| Prediction | Actual | Error (actual minus prediction) | Squared error |
|---|---|---|---|
| 3.0 | 2.0 | -1.0 | 1.00 |
| 5.0 | 5.0 | 0.0 | 0.00 |
| 2.5 | 4.0 | 1.5 | 2.25 |
The error column follows the formula above, actual minus prediction. Subtracting the other way flips every sign and changes nothing once you square, which is why the convention usually goes unstated.
The squared errors sum to 3.25. Divide by three observations and MSE = 1.083. For the same three points, mean absolute error is 0.833 and root mean square error is 1.041. Notice that MSE is the largest of the three and the only one whose value cannot be read as “hours.”
In code it is a one-liner:
import numpy as np
mse = np.mean((y_true - y_pred) ** 2)sklearn.metrics.mean_squared_error computes the same thing with input validation and optional sample weights.
Why the squaring matters
Errors cannot cancel. A model that is 10 too high on one row and 10 too low on the next has an average error of zero and an MSE of 100. Squaring is what stops the metric from calling that model perfect.
Large errors dominate. The penalty grows with the square of the miss, so the metric is driven by the tail. That is the right behavior when a single bad prediction is genuinely catastrophic, such as an inventory forecast that empties a warehouse. It is the wrong behavior when your tail is mostly mislabeled rows.
Gradients scale with the error. The derivative of one squared residual with respect to the prediction is -2 * (y - y_hat). The size of the correction is proportional to how wrong the prediction was, which is why MSE training pulls hardest toward the points it currently fits worst.
The minimizer is the mean. A model trained on MSE with no useful features converges on the mean of the target. Trained on mean absolute error, it converges on the median. That one fact explains most of the behavioral difference between the two losses on skewed data.
What counts as a good MSE
There is no universal threshold, because MSE is scale dependent. An MSE of 0.02 is terrible if your target ranges from 0 to 0.1 and excellent if it ranges from 0 to 10,000.
The comparison that means something is against a baseline: the MSE you would get by predicting the training mean for every row. That baseline is the variance of the target, and the familiar coefficient of determination is exactly that ratio:
R^2 = 1 - MSE / Var(y)A model whose MSE beats the variance of the target is contributing information. One that does not is worse than a constant. Picking the right comparison point is half of choosing an evaluation metric in the first place.
Where MSE misleads
Label noise gets amplified. Squaring does not know the difference between a hard example and a wrong label. A handful of corrupted targets can move MSE more than hundreds of correct predictions, and the training run will happily contort the model to fit them.
It does not decompose the way people assume. Two models with identical MSE can fail for opposite reasons. Compute MSE per slice before trusting the global number, and pair it with statistical distance metrics when inputs shift.
MSE in LLM and agent systems
Most LLM output is text, so MSE does not transfer to generation quality. It still appears for numeric fields an agent extracts, embedding reconstruction objectives, and tabular models inside the stack. For text, use an LLM as a judge evaluator; for units rather than squared units, see what you need to know about RMSE.
FAQ
What is the formula for MSE?
MSE = (1/n) * sum( (y_i - y_hat_i)^2 ), summed over all n observations. Subtract each prediction from its actual value, square the result, and take the average. Some texts write it as SSE divided by n, where SSE is the sum of squared errors. It is the same quantity.
What is the difference between MSE and RMSE?
RMSE is the square root of MSE. They rank models identically, because the square root is monotonic, so whichever model has the lower MSE also has the lower RMSE. The difference is readability: RMSE is in the same units as the target, so it can be quoted as “off by about 1.04 hours,” while MSE cannot.
Is a lower MSE always better?
On the same target and the same dataset, yes. Across different targets, no, because MSE is scale dependent and cannot be compared between models predicting different quantities. A very low MSE on training data with a much higher MSE on held-out data is overfitting, not quality.
Why does MSE penalize outliers so heavily?
Because the penalty is quadratic. Doubling an error quadruples its contribution. That is useful when large misses are genuinely worse than small ones, and harmful when your extreme values are noise. Mean absolute error, which penalizes linearly, is the usual alternative when you do not want the tail steering the model.
Can MSE be negative?
No. Every term in the sum is a square, so MSE is always zero or positive. If a library reports a negative value you are almost certainly looking at a scoring convention such as neg_mean_squared_error, which flips the sign so that higher is better for search routines that maximize.
