Arize:Observe Unstructured, our first-ever summit dedicated to helping machine learning teams take their unstructured data initiatives to the next level, wrapped up last week before an audience of nearly 500 technical leaders and practitioners. Now available for on-demand streaming, discussions from the event continue in the Arize community with speakers from Hugging Face, OpenAI, the Tutte Institute of Mathematics, Labelbox, and Pachyderm continuing to answer questions.
In case you missed it, here are four top takeaways from the event.
Monitoring of Unstructured Data Has Arrived
In the keynote, Arize’s co-founders – CEO Jason Lopatecki and Chief Product Officer Aparna Dhinakaran – formally announced that Arize’s support for embedding analysis and embedding drift monitoring is now available as part of Arize’s free subscription tier.
The new release enables teams to log models with both structured and unstructured data to Arize for monitoring. By monitoring embeddings, teams can proactively identify when their unstructured data is drifting. Interactive, three-dimensional Uniform Manifold Approximation and Projection (UMAP) views help teams quickly visualize new patterns and export similar clusters for continuous model improvement.
“Cutting edge deep learning still relies on human labeling teams looking at around one percent of the data to help train models and – hopefully – capture what will happen in the real world,” notes Dhinakaran. “Better monitoring is needed to surface issues with models in production. Arize’s new capabilities for monitoring unstructured data promise to change the game for ML teams with computer vision and NLP models in production.”
What Works In Training May Not Work In Production When Deciding What To Label Next
In a session on “How To Improve Performance of Unstructured Models with Less Data,” Maxime Voisin, product lead at training data platform Labelbox, offers a primer on how to determine which data to label in training – covering techniques like surfacing low-confidence predictions and leveraging error analysis – before turning his attention to the last mile.
“In terms of when to do additional labeling, I think it’s slightly different when you’re developing your machine learning system and when the model is in production,” Voisin says. “When you’re training your model, you typically have a fixed dataset,” he notes, but “when your models are in production, it’s entirely the opposite — you’re facing a constant stream of data because your models are making predictions all day long, and that blows up really fast and you can’t do things manually anymore.”
As a result, it is critical to “diversify the skills” of the team with an eye toward production, Voisin notes, and to “monitor every single piece in your system and monitor the drift in the data, the drift in the predictions, the state of the different steps of the pipelines to be able to send alerts when something goes wrong.”
Ultimately, most in the industry are “going from a model-centric approach to a data-centric approach and focusing on improving the data,” Voisin observes. “And when you’re improving the data, it turns out that there aren’t many things you can do — you can fix labels, or label more — so you have to be really smart about what you label.”
The Rise of the Single, Unified Model Will Change MLOps
In “Powering the Next Generation of Products with AI,” Peter Welinder, VP of Product & Partnerships at OpenAI, outlines a vision of the future where models trained to do a wide array of tasks supplant highly specialized models. “Combining modalities into a single model is where the field is going,” he notes, citing not just OpenAI’s GPT-3 but also PaLM, Gato, and Google Brain as examples.
The consequences of this trend will likely be far-reaching in several ways. First, it will require new investment to make these more complex models easier to implement and use. “It used to be that everyone trained their own models from scratch, but now many are using open source models and are fine tuning,” notes Welinder. “That’s why we are so focused so much on making it easy to build on top of these models. It’s quite complex to get these models to run fast at scale, and that’s where we focus — making sure it’s reliable and fast and cost effective to make it easier for people.”
The rise of the fewer, unified models will also likely require “a new way of programming,” Welinder continues. “It used to be that you would train models with positive and negative examples. Now, it’s how you explain what you want the model to do. Prompt/assign is the concept…explaining to a model what you want to do well.”
Cutting-Edge Machine Learning Is Becoming More Accessible
In his session on “Accelerating Machine Learning from Research to Production with Hugging Face,” Jeff Boudier, Product Director at the popular community and data science platform, makes the case that adoption of transformer models will accelerate over the next few years through greater accessibility, community collaboration, and reduced costs.
“Because transfer learning has changed the field and is now producing the best possible performance across all of these modalities, it’s the first time that everybody within the AI community is focused on the same set of tools,” notes Boudier. “That’s what makes Hugging Face as the center of that ecosystem so valuable for the community.”
There is work to do before transformer models can enter the mainstream, however. “Transformers are massive models trained on a gigantic corpora of data using a massive amount of computing, and the output is models that would not fit on commodity hardware,” Boudier observes. “It’s a challenge for enterprises today to take those models created with Hugging Face transformers and deploy them very efficiently. That’s a problem we are addressing through our open source and our commercial solutions.”
Potential applications and use cases are exciting, and not just for computer vision or NLP models. “One trend that has been very clear over the past nine months is the extension of the applicability of transformer models based on attention mechanisms to new use cases,” Boudier continues, noting their usefulness in time series, recommender systems, biology, chemistry, and reinforcement learning.
Learn More and Start Your Journey
Want to learn more about unstructured data? Read about monitoring unstructured data and embedding drift monitoring and see why getting started with embeddings is easier than you think. Want to test out some of these techniques yourself? Sign up for a free Arize account, and follow along with our colabs.