AI ROI: Guide To Observability Value Statistics

Introduction

Due to its unique ability to preemptively detect and fix model issues that may be impacting business value, model observability initiatives often yield a high return on investment (ROI).

At Arize, we think of a “model insight” as an issue with a model or data that has an impact on the model performance in terms of accuracy and/or business value. In Arize, a user can automatically detect and root cause a model insight, alerting them to the issue proactively, and then easily trace the issue to a root cause so it can be resolved.

Without the proper observability tools or strategies, model failures go undetected, or can go unnoticed for weeks or longer before they are detected and root-caused manually.

Insights can be automatically detected through Arize monitors. Then, the insights are root-caused through data exploration in Arize in interactive and guided workflows such as UMAP, drift, performance tracing, and explainability tools. Here, the user is guided towards uncovering the most impactful and meaningful trends in their models.

Methodology

The analysis below was conducted over a sample of 50 teams with at least one model in production (many of whom have multiple models). The study spanned 500+ models with varying use cases across companies of various sizes.

Statistics for Initial Model Insights

Once the data for a model is ingested into Arize, uncovering initial model insight is fast through interactive guided workflows in Arize.

• Model insight prevalence: 95% of teams can find a valuable insight when first exploring their data in Arize.
• Time to first insight: Users uncover an initial insight within the first 24 hours of exploring their model data in Arize.

Statistics for Monitoring

Models will naturally have issues over time. The number of problems largely depends on the complexity of the model, data, and deployment infrastructure.

Once proper monitoring coverage is established, model insights can be detected automatically and then root caused in Arize.

On average:

• Confirmed model insights: We see teams detecting confirmed model issues with Arize monitors once a month.
• Time to detect: Detection is immediate with Arize monitoring.
• Time to root cause: Users can root cause a monitor alarm within the first 24 hours through exploring their model data in Arize. Without ML observability, this can take days to weeks.
• Time to resolve: Resolution depends on the remedial step required. Often, automated model retraining will resolve many model issues. Some issues will require additional data collection or labeling, and model experimentation to resolve.

There is an art to configuring the right monitors for a model and it is highly dependent on model type, feedback loop data available, and feature set. Arize offers training and guides on Monitoring best practices (feel free to reach out in the Arize community for help).

Observability Cost Savings

Often, model insights and improvements can and should be correlated back to business metrics to show ROI and cost savings and observability initiatives. This breakdown shows the value of ML observability per model, based on the estimated cost of productivity and business value of catching model problems in production.

What Is AI ROI?

AI ROI is a method of measuring the value of an AI project to a business. Good AI ROI metrics quantify the impact of modeling projects, model accuracy, and model improvements in terms of lift to key business metrics. Quantifying AI project ROI will enable easy calculations of the value of observability and individual insights for each project and model.

Here are some examples of great AI ROI on data science projects.

• “A data scientist developed a ML model for product recommendations that increased sales by over 25%.”
• “A data scientist developed a ML model for ad targeting that increased revenue by 20%.”
• “A data scientist developed a model to optimize delivery routes that reduced delivery time by 20% and saved the company $500,000 annually in transportation costs.”
• “A data scientist developed an ML-powered chatbot that reduced response time by 50% and improved customer satisfaction scores by 10%.”
• ” A data scientist developed an ML model for fraud detection that reduced fraud rates by 50% and saved the company $2 million annually in fraud charges.”

While AI ROI is not prescriptive and every business will have a different set of metrics they care to optimize. Arize custom metrics allows the user to define any metric using model data and metadata, and track these personalized metrics in Arize monitors and dashboards.

Conclusion

As the adoption of AI accelerates, technical teams need to be able to consistently quantify the ROI of AI initiatives. Whether it’s a chatbot or recommendation system, having an observability strategy is a key first step in that process.