Bazaarvoice helps ecommerce brands manage and make sense of user-generated content, including ratings, reviews, images, and video. The company has used AI extensively for years and now sees generative AI as another tool for building new products and customer experiences.
As Bazaarvoice expanded its use of generative AI, the team needed a faster way to understand whether LLM-powered experiences were actually producing high-quality results. Evals quickly became a practical part of that workflow, helping measure quality at scale without relying on manual review for every output.
Challenge: Evaluating AI against decades of real customer data
One of the experiences Bazaarvoice is developing is a question-and-answer system built on consumer-generated content.
Imagine a shopper considering a pair of shoes. Instead of reading through dozens of reviews, they can ask a question about the product and receive an AI-generated answer based on what previous customers have shared.
For Bazaarvoice, evaluating those answers comes with an advantage: decades of historical data containing real questions from consumers and answers provided by brands.
“We have decades worth of data of example questions consumers have asked and actual answers that our clients have provided to them,” Kratz explains. “So we can directly compare the answers that our clients wrote to ones generated by the AI system.”
This gives Bazaarvoice a real-world benchmark. But comparing AI-generated answers against that benchmark at scale creates a problem: manually reviewing thousands or millions of outputs quickly becomes slow and expensive.
The team needed a way to answer a fundamental question at scale: How often is the AI correct, and how often is it wrong?
The shift: Use LLMs to evaluate LLMs
Instead of relying on people to manually review every AI-generated answer, Bazaarvoice began using LLM-based evals to classify outputs at much greater scale.
The approach turns evaluation into a simpler problem. Rather than asking an LLM to generate another answer, the evaluator focuses on a narrower task, a classifier rather than a generator. This distinction is critical: using one LLM to evaluate another works specifically because evals are asking the model to classify, not generate.
“You’re asking an AI model to evaluate another AI model, and the only reason this is working is that the evals themselves are a much simpler task in that they are a classifier and not a generator,” Lou says.
For Bazaarvoice, examples include determining whether two answers effectively say the same thing or classifying an output as correct or incorrect.
That doesn’t eliminate human judgment. Instead, it changes where people spend their time. Humans establish a trusted set of labeled examples, benchmark the evaluator against those examples, and then use the validated evaluator across a much larger dataset.
Result & impact: From a day and a half of labeling to 10 minutes
Initially, answering the accuracy question required people to manually review and label AI-generated outputs.
For one experiment, Bazaarvoice sent 1,000 examples for human review. The process took about a day and a half and came with the associated cost of manual labeling.
With LLM-based evals, Kratz says the same volume can be evaluated in about 10 minutes.
“We sent them a thousand examples and it took them a day and a half, and it cost us a bunch of money,” Lou says. “With an eval, I can do that same amount in about 10 minutes and it costs us pennies.”
That’s a tiny fraction of the manual labor cost and the difference compounds as you scale evaluation across larger datasets. Instead of relying on people to manually review every output, the team can use LLMs to classify much larger volumes of data as correct or incorrect and quickly understand how the system is performing.
“It’s saving us a lot of time on my team.”
Principal Engineer, Bazaarvoice
Lessons: Building evals teams can trust
Human judgment still matters
Using an LLM to evaluate another LLM introduces its own challenge: How do you know the evaluator is right? Bazaarvoice addresses this by benchmarking its evals against known, human-labeled results.
“You’ve created a second model to grade your first model,” Lou explains. “You have to evaluate the accuracy of that second model too.”
The difference is that the team doesn’t need to manually label every output. Instead, it can invest human effort in a smaller test dataset, determine whether the eval reliably matches those judgments, and then apply the eval across a much larger dataset.
“I only need to label maybe a thousand pieces of responses for my eval to know my eval is working, and then I can use it to go measure my system on a million pieces of content.”
For Bazaarvoice, human evaluation isn’t disappearing. It’s being used more strategically to validate the system that makes evaluation scalable.
Keep evals simple
Another lesson from Bazaarvoice’s work is that effective evals don’t need to be complicated.
The team primarily uses categorical evaluations, such as determining whether an answer is correct or incorrect, along with multiclass evaluations when there is more nuance in quality. The key is keeping each evaluator focused on a clearly defined task.
One practical detail: different LLMs have different prompt preferences. When Bazaarvoice ported prompts from GPT to Claude, they discovered Claude prefers XML formatting over JSON or Markdown. Understanding your model’s preferences upfront saves significant debugging time later.
Rather than building a single evaluator that tries to measure multiple dimensions of quality at once, Lou recommends starting with one clear evaluation, benchmarking it, and then adding another.
“Keep it simple and straightforward,” he says. “You got one working, make the next one.”
What’s next: Building an evaluation loop that gets better with production
As Bazaarvoice’s AI systems move toward production, the team’s evaluation datasets can continue evolving with them. Real-world failures can become new test cases. Those examples can be added back into evaluation datasets, helping the team test future versions against problems that actually occurred.
Over time, this creates a feedback loop between what happens in production and what the team tests next. For Lou, the lesson is less about finding the perfect evaluation framework from day one and more about building one that can improve alongside the AI system.
Start with a clear question. Create a small set of trusted human labels. Benchmark the evaluator against them. Then use that evaluator to understand quality across much larger datasets.
The result is a workflow that lets Bazaarvoice spend less time manually labeling AI outputs and more time understanding and improving the experiences it is building.
