How Bazaarvoice Navigated the Challenges of Deploying an LLM App
Sarah Welsh
Contributor
Bazaarvoice, a top platform for user-generated content and social commerce, has leveraged AI for much of its history — and now has a pioneering LLM app in production.
At Arize:Observe this year, we caught up with Lou Kratz, Principal Research Engineer at Bazaarvoice, to talk about how their team successfully worked through the challenges that went along with that.
He named two challenges that you may not want to neglect when getting an LLM app into production.
The first is data quality for RAG: “You look at something like retrieval augmented generation — it’s really powerful, it can really make things explainable and usable to the general public — but it’s only as good as the data we give it. When it comes to business-specific data, the first challenge is getting that cleaned up.”
The second is around education: “Almost all of our data scientists and engineers have become mentors…in order to help people understand the specifics about how AI works and if it solves their use case.”
Catch the full conversation with Lou Kratz below.
Watch it
Tell us about your role at Bazaarvoice
I’m Lou Kratz, a Principal Engineer at Bazaarvoice. I specialize in AI, so I lead all the AI development efforts from a technical perspective, as well as the technical lead of the team. Today I’m presenting on evals and how we use them to save our machine learning engineers and data scientists time, as well as reduce the cost of developing new models.
What are your Gen AI use cases?
Our use cases at Bazaarvoice are pretty wide. We’ve been around about 10 years, and have been doing AI for most of that. We have models ranging from decision trees to convolutional neural nets and now to generative models using prompts. For all of those, we use Arize and Phoenix to measure their outcomes as well as the availability.
So our big use case in Arize was a little bit around observability, but it was really around being able to show the value that our AIs bring to the business by reporting outcome statistics into Arize. So even non technical folks can go on and see those dashboards and say, hey, that model has made us this much money this year, or this client isn’t doing as well there, and get those insights without having to ask an engineer to dig deep into the data.
What are the challenges in getting an LLM app into production?
With LLMs, I’d say there’s two big challenges. One is getting people to kind of clean up their data. Everyone’s really excited about what LLMs can do for internal and external use cases. You look at something like retrieval augmented generation, it’s really powerful, it can really make things explainable and usable to the general public, but they’re only as good as the data we give it.
When it comes to business specific data, the first big challenge we’ve faced is getting that cleaned up. We’re still overcoming that, but AI is great because it gets people excited so they can try it out with their data and realize they need to invest in their data in order to get the most out of it. So, it’s really been an accelerant to get people to clean up their data.
The second big problem I’ve found is really around education. A lot of folks, even engineers may not know how it works or the limitations thereof. So almost all of our data scientists and engineers have become mentors or leaders in the company in order to help people understand the specifics about how AI works and if it can solve their use case when they come to us with ideas.
How is AI transforming Bazaarvoice’s business?
The biggest impact AI has at Bazaarvoice is around ensuring the content that we provide our clients, which are generated by users, is of high quality. And when I say high quality, I don’t necessarily mean a positive review, I mean a review that really says what’s going on.
So we’ve used generative AI recently to release what we call a content coach that guides consumers through the process of writing a good review. Historically, we did similar things using AI and ML to moderate our content to make sure harmful content doesn’t get on our client’s website. And we’re currently experimenting with new computer vision models and things like that to try to get better quality images as well.