While the industry has invested a lot in processes and techniques for knowing when to deploy a model into production, there is arguably less collective knowledge on the equally important task of knowing when to retrain a model. In truth, knowing when to retrain a model is hard due to factors like delays in feedback or labels for live predictions. In practice, many practitioners just end up training on a specific schedule — or not at all — and hope for the best.
Based on direct experience working with customers with models in production topping billions of daily predictions, this guide covers:
- How to shift to automated, dynamic model retraining
- Best practices for metrics-driven retraining and strategies for promoting a new model and navigating retraining feedback loop data
- How to calculate AI ROI