ML Observability Advanced Metrics Course

Welcome to Arize University’s ML Observability Advanced Metrics Certification Course! This self-guided course will teach you intuition, math, and best practices for effective machine learning observability across a variety of use cases. Learn how to calculate the most common metrics in model monitoring – including performance, drift, ranking and fairness metrics. We dive into performance metrics for classification, regression and ranking models, as well as the prevailing statistical distance measurements, as well as parity metrics, in detail and practical advice on where to use each.

This course attempts to do three things:

Provide an intro to the most popular performance, drift and fairness metrics in data science
Give insights and intuition around their use, based on practical experience
Describe how to use and monitor these metrics in production environments

Enroll now

University 3D

Arize University Courses

ML Observability Fundamentals
Click here →

Advanced Metrics Course
(you're in the right place!)

What to expect

Total completion time is 2–3 hours, can be taken in parts

Is best taken after completing Arize’s ML Observability Fundamentals course

Upon course completion you will receive a Linkedin certification badge

Course Curriculum

Introduction ML Observability Advanced Metrics Overview

Get your advanced metrics certification started with intuition, math, and best practices for effective machine learning observability. Learn how to calculate and use some of the most common metrics in model monitoring – including for performance, ranking, drift, and fairness metrics.

<img width="31" height="31" data-src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 31px; --smush-placeholder-aspect-ratio: 31/31;"><noscript><img width="31" height="31" data-src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 31px; --smush-placeholder-aspect-ratio: 31/31;"><noscript><img width="31" height="31" data-src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 31px; --smush-placeholder-aspect-ratio: 31/31;"><noscript><img width="31" height="31" src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650">

Course video (7 minutes)

<img width="27" height="30" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 27px; --smush-placeholder-aspect-ratio: 27/30;"><noscript><img width="27" height="30" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 27px; --smush-placeholder-aspect-ratio: 27/30;"><noscript><img width="27" height="30" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 27px; --smush-placeholder-aspect-ratio: 27/30;"><noscript><img width="27" height="30" src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046">

Quiz (5 minutes)

<img width="29" height="28" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 29px; --smush-placeholder-aspect-ratio: 29/28;"><noscript><img width="29" height="28" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 29px; --smush-placeholder-aspect-ratio: 29/28;"><noscript><img width="29" height="28" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 29px; --smush-placeholder-aspect-ratio: 29/28;"><noscript><img width="29" height="28" src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063">

Lab (2 minutes)

Enroll now →

Unit 1 Performance Metrics in Production

The idea of model performance management encompasses managing model performance across the full lifecycle from testing through production. This unit covers the top performance metrics used by AI teams to monitor their models in production, as well as breaking down changes in model performance.

<img width="31" height="31" data-src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 31px; --smush-placeholder-aspect-ratio: 31/31;"><noscript><img width="31" height="31" data-src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 31px; --smush-placeholder-aspect-ratio: 31/31;"><noscript><img width="31" height="31" data-src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 31px; --smush-placeholder-aspect-ratio: 31/31;"><noscript><img width="31" height="31" src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650">

Course video (34 minutes)

<img width="27" height="30" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 27px; --smush-placeholder-aspect-ratio: 27/30;"><noscript><img width="27" height="30" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 27px; --smush-placeholder-aspect-ratio: 27/30;"><noscript><img width="27" height="30" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 27px; --smush-placeholder-aspect-ratio: 27/30;"><noscript><img width="27" height="30" src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046">

Quiz (7 minutes)

<img width="29" height="28" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 29px; --smush-placeholder-aspect-ratio: 29/28;"><noscript><img width="29" height="28" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 29px; --smush-placeholder-aspect-ratio: 29/28;"><noscript><img width="29" height="28" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 29px; --smush-placeholder-aspect-ratio: 29/28;"><noscript><img width="29" height="28" src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063">

Lab (3 minutes)

Enroll now →

Unit 2 Ranking Metrics in Production

Ranking models underpin many aspects of modern digital life, from search results to music recommendations. However, personalized ranking and recommendation engines are notoriously difficult to maintain in production, this unit will advise you on how to properly monitor and maintain performance in ranking systems.

<img width="31" height="31" data-src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 31px; --smush-placeholder-aspect-ratio: 31/31;"><noscript><img width="31" height="31" data-src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 31px; --smush-placeholder-aspect-ratio: 31/31;"><noscript><img width="31" height="31" data-src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 31px; --smush-placeholder-aspect-ratio: 31/31;"><noscript><img width="31" height="31" src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650">

Course video (14 minutes)

<img width="27" height="30" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 27px; --smush-placeholder-aspect-ratio: 27/30;"><noscript><img width="27" height="30" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 27px; --smush-placeholder-aspect-ratio: 27/30;"><noscript><img width="27" height="30" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 27px; --smush-placeholder-aspect-ratio: 27/30;"><noscript><img width="27" height="30" src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046">

Quiz (7 minutes)

<img width="29" height="28" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 29px; --smush-placeholder-aspect-ratio: 29/28;"><noscript><img width="29" height="28" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 29px; --smush-placeholder-aspect-ratio: 29/28;"><noscript><img width="29" height="28" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 29px; --smush-placeholder-aspect-ratio: 29/28;"><noscript><img width="29" height="28" src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063">

Lab (3 minutes)

Enroll now →

Unit 3 Drift Metrics in Production

For ML models without fast actuals, measuring drift is the best proxy to understand model performance. Learn how to calculate the prevailing statistical distance measurements in detail and which use cases to apply them.

<img width="31" height="31" data-src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 31px; --smush-placeholder-aspect-ratio: 31/31;"><noscript><img width="31" height="31" data-src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 31px; --smush-placeholder-aspect-ratio: 31/31;"><noscript><img width="31" height="31" data-src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 31px; --smush-placeholder-aspect-ratio: 31/31;"><noscript><img width="31" height="31" src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650">

Course video (26 minutes)

<img width="27" height="30" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 27px; --smush-placeholder-aspect-ratio: 27/30;"><noscript><img width="27" height="30" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 27px; --smush-placeholder-aspect-ratio: 27/30;"><noscript><img width="27" height="30" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 27px; --smush-placeholder-aspect-ratio: 27/30;"><noscript><img width="27" height="30" src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046">

Quiz (7 minutes)

<img width="29" height="28" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 29px; --smush-placeholder-aspect-ratio: 29/28;"><noscript><img width="29" height="28" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 29px; --smush-placeholder-aspect-ratio: 29/28;"><noscript><img width="29" height="28" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 29px; --smush-placeholder-aspect-ratio: 29/28;"><noscript><img width="29" height="28" src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063">

Lab (3 minutes)

Enroll now →

Unit 4 Fairness Metrics in Production

Model fairness impacts the pre-processing, in-processing, and post-processing stages of the data modeling pipeline. Learn how to evaluate fairness through parody metrics at each stage of this process to ensure that the model is not biased against certain groups.

<img width="31" height="31" data-src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 31px; --smush-placeholder-aspect-ratio: 31/31;"><noscript><img width="31" height="31" data-src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 31px; --smush-placeholder-aspect-ratio: 31/31;"><noscript><img width="31" height="31" data-src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 31px; --smush-placeholder-aspect-ratio: 31/31;"><noscript><img width="31" height="31" src="https://arize.com/wp-content/uploads/2022/09/Path-12650.svg" alt="Video icon" title="Path 12650">

Course video (14 minutes)

<img width="27" height="30" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 27px; --smush-placeholder-aspect-ratio: 27/30;"><noscript><img width="27" height="30" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 27px; --smush-placeholder-aspect-ratio: 27/30;"><noscript><img width="27" height="30" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 27px; --smush-placeholder-aspect-ratio: 27/30;"><noscript><img width="27" height="30" src="https://arize.com/wp-content/uploads/2022/09/Group-6046.svg" alt="Quiz" title="Group 6046">

Quiz (7 minutes)

<img width="29" height="28" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 29px; --smush-placeholder-aspect-ratio: 29/28;"><noscript><img width="29" height="28" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 29px; --smush-placeholder-aspect-ratio: 29/28;"><noscript><img width="29" height="28" data-src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063" src="data:image/gif;base64,R0lGODlhAQABAAAAACH5BAEKAAEALAAAAAABAAEAAAICTAEAOw==" class="lazyload" style="--smush-placeholder-width: 29px; --smush-placeholder-aspect-ratio: 29/28;"><noscript><img width="29" height="28" src="https://arize.com/wp-content/uploads/2022/09/Group-6063.svg" alt="Lab" title="Group 6063">

Lab (3 minutes)

Enroll now →

Technical requirements

Python

Mathematics

Earn your certificate in only a few hours

Enroll now

Arize University certificate-advanced

Completion requirements

Upon completing all units of the ML Observability Advanced Metrics Course, you will receive a Certificate of Completion to highlight your new skillset. Completion of all the unit labs, as well as the passing of each unit quiz are required for completion.

Join the Arize Machine Learning Observability Course

Enroll for free