What is Prediction Drift?
Prediction drift (aka model drift) represents a change in a model’s predictions over time. Prediction drift also reflects a change in predictions from new values compared to pre-production predictions. Once you know what exactly is causing the model or prediction drift, it’s possible to combat it by retraining the model with additional data or replacing the model.
Why Is Monitoring For Prediction Drift Important?
Assuming your model is deterministic, and nothing in your feature pipelines has changed, your model should give the same results if it sees the same inputs. Yet, in production, models decay over time. Monitoring for prediction drift provides insights into model quality and overall model performance.
While it would be ideal if inputs were the only thing to change in your model, and everything else remains stagnant, unfortunately, that’s not the case when a model is in production, and even if your model is making the same predictions as yesterday, it can make mistakes today!
Prediction drift can point to a degradation in your model’s performance and it’s important to catch prediction drift before your model degrades to the point of negatively impacting your customers’ experience or intended business outcomes.
Paper explores types of data set shift in classification, including covariate shift, prior probability shift, and concept shift.
Paper explores ODIN, a visual data analytics system designed to automatically detect and recover from drift.
A guide to model failure modes and detecting, diagnosing, and explaining regressions models that have been deployed to production. Explores model drift with examples.
Learn about ML observability, which compares distribution changes between a baseline distribution and a current distribution and how model owners can do targeted upsampling when there is drift.
A quick guide with basketball metaphors on the different types of drift and what drift means in the context of ML observability
Use cases for statistical distance checks and model drift across model inputs, model outputs and actuals.