Introduction
Welcome! This self-guided course will teach you some of the intuition, math, and best practices for effective machine learning observability. Learn how to calculate and use some of the most common metrics in model monitoring – including for performance, drift, data quality, fairness, and service-level performance – as well as the latest techniques in monitoring embeddings and unstructured data.
What To Expect?
This course:
- Includes math and Python
- Can be taken in parts (each lesson is its own self-contained primer)
- Is best taken after completing or in tandem with our ML Observability Fundamentals certification course
Here is a quick overview of the course: After a quick overview on observability, Model Evaluation Metrics covers the intuition, math, and best practices in model monitoring and performance tracing for prevailing metrics across different use cases and model types. Similarly, Drift Metrics covers the prevailing statistical distance measurements in detail and practical advice on where to use each. Additional sections on Data Quality, Fairness Metrics, Explainability, and Service Monitoring Metrics round out the course. Finally, an (expanding) section on Unstructured Data and Embeddings dives into the latest techniques for visualizing embeddings, monitoring vector drift, and more.
Machine Learning Certifications
Ready to get certified and prove you have the knowledge? Arize University offers a parallel course with a machine learning certification specializing in ML observability. Check it out!
Who Are We?
About the authors of the course:
Jason Lopatecki is Co-Founder and CEO of Arize AI. He is a garage-to-IPO executive with an extensive background in building marketing-leading products and businesses that heavily leverage analytics. Prior to Arize, Jason was co-founder and chief innovation officer at TubeMogul, where he scaled the business into a public company and eventual acquisition by Adobe. Jason has hands-on knowledge of big data architectures, programmatic advertising systems, distributed systems, and machine learning and data processing architectures. In his free time, Jason tinkers with personal machine learning projects as a hobby, with a special interest in unsupervised learning and deep neural networks. He holds an electrical engineering and computer science degree from UC Berkeley.
Aparna Dhinakaran is the Co-Founder and Chief Product Officer at Arize AI. A frequent speaker at top conferences and thought leader in the space, Dhinakaran was recently named to the Forbes 30 Under 30. Before Arize, Dhinakaran was an ML engineer and leader at Uber, Apple, and TubeMogul (acquired by Adobe). During her time at Uber, she built several core ML Infrastructure platforms, including Michelangelo. She has a bachelor’s from UC Berkeley’s Electrical Engineering and Computer Science program, where she published research with Berkeley’s AI Research group. She is on a leave of absence from the Computer Vision PhD program at Cornell University.
Amber Roberts is a community-oriented machine learning engineer at Arize AI. Amber’s role at Arize looks to help teams across all industries build ML Observability into their productionalized AI environments. Previously, Amber was a product manager of AI at Splunk and the Head of Artificial Intelligence at Insight Data Science. A Carnegie Fellow, Amber has an MS in Astrophysics from the Universidad de Chile. When Amber isn’t expertly teaching ML observability best practices, you can find Amber playing with her two puppies, Rusty and Sully, on Florida’s warm beaches.
Francisco Castillo Carrasco (“Kiko”) is a data scientist and engineer at Arize AI. He has an MA in applied mathematics from Arizona State University and previously did research at the von Karman Institute for Fluid Dynamics.
Jianshu Chi is a Software Engineer at Arize AI. Previously, he was a researcher at UC Berkeley. There, he did research in machine learning and discovered the first in vivo human brain and hand imaging using two-photon excitation technique (MRI). A former executive chef, Jianshu has an MS in Computer Science from UC Berkeley.