What is an Evaluation Store?

Evaluation Store

An evaluation store -- also sometimes referred to as an inference store -- is a machine learning infrastructure tool used to monitor and improve model performance. Think of them as the ledger or log of model activities/inferences. Evaluation stores are used to:

Surface up performance metrics in aggregate (or slice) for any model, in any environment — production, validation, training
Monitor and identify drift, data quality issues, or anomalous performance degradations using baselines
Enable teams to connect changes in performance to why they occurred
Provide a platform to help deliver models continuously with high quality and feedback loops for improvement — compare production to training
Provide an experimentation platform to A/B test model versions