Mean Absolute Error is a regressive loss measure looking at the absolute value difference between a model’s predictions and ground truth, averaged out across the dataset. Unlike MSE, MAE is weighted on a linear scale and therefore doesn’t put as much weight on outliers. This provides a more even measure of performance, but means large errors and smaller errors are weighted the same. Something to consider depending on your specific model use case.
Mean Absolute Error (MAE) is a commonly used metric for evaluating the accuracy of predictions. It measures the average absolute difference between the actual and predicted values. It is used when the goal is to evaluate the quality of predictions in terms of their absolute magnitude, rather than their relative magnitude.
A possible mitigation may be to use the log cosh loss function, which is similar to the MAE but twice differentiable.
Regression Models
Before diving into mean absolute error, it helps to have some preliminary context on regression models.
Regression is among the most widely-used techniques in supervised machine learning, popular for everything from quantitative finance to demand forecasting since it straightforwardly captures a linear relationship between two or more variables.
Let’s walk through an example: building a linear regression for housing prices where the only feature is the house’s square footage. Our graph would have square footage on the x-axis and housing price on the y-axis, like this:
Then, we would fit a line that captures the linear relationship between the square footage and the housing price. Now we can make a price prediction for every house with a square footage value, or in other words every x is predicted to have the corresponding y value.
This linear relationship is only an approximation so some of the square footage and price pairings will be spread out around the line fit by our model. One reason that engineers use MAE in practice is that it is robust to outliers, but there are still errors associated with how far off the line of best fit is from the actual value.
In order to calculate MAE, you must subtract each of the ground truths (actuals) from the corresponding value on the prediction line – this is known as the residual and it represents our model’s error for that particular data point.
The major regression metrics are primarily distinguished from each other based on what they do with these residuals.
Mean Absolute Error Definition
The mean absolute error is the average of the absolute values of a model’s errors.
Where:
- n is the number of fitted points,
- xi is the prediction,
- x is the actual.
To calculate the mean absolute error, we first calculate the absolute value of all the residuals. We take the absolute value of the errors because we don’t want positive and negative errors to cancel each other out. If our model overshoots one data point by +10 and undershoots another by -10, these errors would cancel each other out because (-10) + 10 = 0. But by taking the absolute value of the errors we avoid this problem, because |10| + |-10| = 20.
Once we calculate the absolute value of all the residuals, we add them all up and divide them by the total number of errors. This gives us the mean absolute error for our regression model.
When Should You Use Mean Absolute Error?
MAE is a regressive loss measure looking at the absolute value difference between a model’s predictions and ground truth, averaged out across the dataset. Unlike Mean Squared Error (MSE), MAE is weighted on a linear scale and therefore doesn’t put as much weight on outliers. This provides a more even measure of performance, but means large errors and smaller errors are weighted the same. Something to consider depending on your specific model use case.
Mean Absolute Percentage Error (MAPE) is one of the most common metrics of model prediction accuracy and the percentage equivalent of Mean Absolute Error (MAE), so while MAPE is a relative measure of error, MAE is linear measure of error.
Monitoring Mean Absolute Error
The Arize platform makes it easy to monitor the mean absolute error. First, your model is used to make predictions, which are then collected and given an ID. These predictions are grouped by a time period, which is usually daily. The ground truth – actuals, or correct data from the real world – is linked to the predictions based on prediction ID. From this, we can calculate the mean absolute error.
Generating the ground truth data can involve delays of days, weeks, or months, which means there might be some instability in the mean absolute error. Also, be wary of cases where there are only a small number of daily predictions. If you only receive a small number of predictions or ground truth in a given day, the metric may end up having a large variance. One possible workaround is bootstrapping the sample in the window to help visualize and understand the metric’s variance.
Conclusion
With regression models being such a popular staple in machine learning work, it’s critical that you understand not just the broad differences in the major regression metrics, but also the subtleties in applying them in real-world cases. Hopefully this post gives you some practical advice on using mean absolute error.