Model Drift

Your one-stop shop for all things model drift-related. Learn what constitutes model drift, how to monitor for drift in ML models, and drift resolution techniques for models with or without actuals.

What is model drift?

Drift is a change in distribution over time, measured for model inputs, outputs, and actuals of a model. Measure drift to identify if your models have grown stale, you have data quality issues, or if there are adversarial inputs in your model. Detecting drift in your models will help protect your models from performance degradation and allow you to better understand how to begin resolution.

Ground Truth
arrow icon
background image
Model Drift/
Concept Drift
Outputs
background image
Model Drift/
Concept Drift
Explainability
(SHAP)
arrow icon
image model box
background image
Model Drift/
Concept Drift
Inputs
background image
Data Drift/
Feature Drift

Types of model drift

Monitoring for model drift falls into three categories to ensure high performing models in production

Feature Drift Change in Distributions

Jump to section →
background gradient

Prediction Drift Change in Relationships

Jump to section →
background gradient

Concept Drift In the case of actuals

Jump to section →
background gradient

Model Drift Metrics No actuals

Jump to section →
background gradient
Feature Drift

When evaluating model drift, monitoring for feature drift helps ML practitioners understand change associated with the inputs of a model. Inputs are not static and feature drift is inevitable; monitoring for feature drift helps catch and resolve performance issues quickly.

CORNELL UNIVERSITY Analysis of Drifting Features Read More read more
MDPI Adaptive Quick Reduct For Drifting Features Read More read more
MIT PRESS Dataset Shift in Machine Learning Read More read more
ARIZE Take my Drift Away Read More read more
ARIZE A Quick Start To Data Quality Monitoring For Machine Learning Read More read more
CORNELL UNIVERSITY Detection of Data Drift and Outliers Affecting ML Models Performance Over Time Read More read more
Prediction Drift

Prediction drift is the change in actuals in a production model. Models decay over time, and monitoring for prediction drift provides insights into measuring model quality over time.

PENN ENGINEERING A Unifying View of Dataset Shift in Classification Read More read more
VLDB Automated Drift Detection and Recovery Read More read more
ARIZE The Model Had Shipped, What Could Possibly Go Wrong? Read More read more
ARIZE Beyond Monitoring: The Rise of Observability Read More read more
VIRGINIA UNIVERSITY Ensemble Learning For Data Stream Analysis: A Survey Read More read more
Concept Drift

Concept drift refers to the changing relationship between the inputs and outputs of a model. Monitoring for concept drift helps ensure models are accurate and relevant in the real world.

CORNELL UNIVERSITY Learning Under Concept Drift: A Review Read More read more
EINDHOVEN UNIVERSITY A Survey on Concept Drift Adaptation Read More read more
CORNELL UNIVERSITY Characterizing Concept Drift Read More read more
CORNELL UNIVERSITY Understanding Concept Drift Read More read more
CITESEERX The Problem of Concept Drift: Definitions and Related Work Read More read more
Model Drift Metrics

Drift is largely measured by comparing the distributions of the inputs, outputs, and actuals between training and production. Model drift metrics are not one-size-fits-all and vary depending on your use case.

NCBI F1

The harmonic mean between precision and recall.

Read More read more
NIST Kolmogorov-Smirnov (K-S) Test

A nonparametric test that compares the cumulative distributions of two data sets. The null hypothesis for this test states that the distributions from both datasets are identical. If the null is rejected then you can conclude that your model has drifted.

Read More read more
KAGGLE Population Stability Index (PSI)

A metric used to measure how a variable’s distribution has changed over time. It is a popular metric used for monitoring model drift as it measures changes in the characteristics of a population, and thus, detecting model decay.

Read More read more
O'REILLY Z-Score

A comparison metric to measure the feature distribution between the training and live data. For example, if a number of live data points of a given variable have a z-score of +/- 3, the distribution of the variable may have shifted.

Read More read more
UNIVERSITY OF ILLINOIS K-L Divergence

A measure of how one probability distribution is different from a second, reference probability distribution, used to detect model drift when one distribution is much smaller in sample numbers and has a large variance.

Read More read more
CARNEGIE MELLON UNIVERSITY Wasserstein’s Distance

A distance function defined between probability distributions on a given metric, best used to detect model drift when there are naturally non-overlapping distributions where KL/PSI need modifications.

Read More read more

Sign up for our monthly newsletter, The Drift.

Subscribe