Knowing when to retrain models in production is hard, due to delays in feedback or labels for live predictions and distributed logic across different pipelines. As a result, users monitor thousands of statistical measures on feature and output spaces and trigger alerts whenever any measure deviates from a baseline, which can lead to many (ignored) false positive signals to retrain models. In this talk, we discuss these challenges in greater detail and propose preliminary solutions to improve precision in retrain alerts.
Speakers
Shreya Shankar
PhD candidate, RISELab UC Berkeley
Shreya Shankar (she/her) is a PhD student at the UC Berkeley RISELab studying data management for production ML systems. Her current research focus is on ML observability. Previously, she was an ML engineer at a startup, did ML research at Google Brain, and completed her BS and MS in computer science at Stanford.
