How Bazaarvoice uses LLM evals to cut AI evaluation from days to minutes

Bazaarvoice helps ecommerce brands manage and make sense of user-generated content, including ratings, reviews, images, and video. The company has used AI extensively for years and now sees generative AI as another tool for building new products and customer experiences.

As Bazaarvoice expanded its use of generative AI, the team needed a faster way to understand whether LLM-powered experiences were actually producing high-quality results. Evals quickly became a practical part of that workflow, helping measure quality at scale without relying on manual review for every output.

“It’s saving us a lot of time on my team,” says Lou Kratz, Principal Engineer at Bazaarvoice.

Evaluating AI against decades of real customer data

One of the experiences Bazaarvoice is developing is a question-and-answer system built on consumer-generated content.

Imagine a shopper considering a pair of shoes. Instead of reading through dozens of reviews, they can ask a question about the product and receive an AI-generated answer based on what previous customers have shared.

For Bazaarvoice, evaluating those answers comes with an advantage: decades of historical data containing real questions from consumers and answers provided by brands.

“We have decades worth of data of example questions consumers have asked and actual answers that our clients have provided to them,” Kratz explains. “So we can directly compare the answers that our clients wrote to ones generated by the AI system.”

The team can use an eval to compare the two answers and determine whether they are effectively saying the same thing. This gives Bazaarvoice a way to measure a fundamental question about the experience: How often is the AI correct, and how often is it wrong?

From a day and a half of labeling to 10 minutes

Initially, answering that question required people to manually review and label AI-generated outputs.

For one experiment, Bazaarvoice sent 1,000 examples for human review. The process took about a day and a half and came with the associated cost of manual labeling.

With LLM-based evals (or LLM-as-a-Judge), Kratz says the same volume can be evaluated in about 10 minutes.

“We sent them a thousand examples and it took them a day and a half, and it cost us a bunch of money,” Lou says. “With an eval, I can do that same amount in about 10 minutes and it costs us pennies.”

That changes what evaluation can look like at scale.

Instead of relying on people to manually review every output, the team can use LLM-as-a-Judge to classify much larger volumes of data as correct or incorrect and quickly understand how the system is performing.

Human judgment still matters

Using an LLM to evaluate another LLM introduces its own challenge: How do you know the evaluator is right?

Bazaarvoice addresses this by benchmarking its evals against known, human-labeled results.

“You’ve created a second model to grade your first model,” Lou explains. “You have to evaluate the accuracy of that second model too.”

The difference is that the team doesn’t need to manually label every output. Instead, it can invest human effort in a smaller test dataset, determine whether the eval reliably matches those judgments, and then apply the eval across a much larger dataset.

“I only need to label maybe a thousand pieces of responses for my eval to know my eval is working, and then I can use it to go measure my system on a million pieces of content.”

For Bazaarvoice, human evaluation isn’t disappearing. It’s being used more strategically to validate the system that makes evaluation scalable.

Keeping agent evals simple

Another lesson from Bazaarvoice’s work is that effective evals don’t need to be complicated.

The team primarily uses categorical evaluations, such as determining whether an answer is correct or incorrect, along with multiclass evaluations when there is more nuance in quality.

The key is keeping each evaluator focused on a clearly defined task.

“You’re asking an AI model to evaluate another AI model, and the only reason this is working is that the evals themselves are a much simpler task in that they are a classifier and not a generator,” Lou says.

Rather than building a single evaluator that tries to measure multiple dimensions of quality at once, Lou recommends starting with one clear agent evaluation, benchmarking it, and then adding another.

“Keep it simple and straightforward,” he says. “You got one working, make the next one.”

That also makes it easier to understand individual dimensions of quality and identify where an AI system needs improvement.

Asking the evaluator to explain itself

One of the most useful changes Bazaarvoice made to its evaluation workflow was also one of the simplest: asking the LLM to explain why it made a decision.

Those explanations give the team more than a label. They provide context for why an output was classified a certain way.

For Lou Kratz, that has become particularly valuable when an eval gets something wrong.

“If I can look at a false positive and understand, well, why did the LLM say that, or if I’m benchmarking the eval and it’s getting it wrong, I can understand why and go and revise my prompt and run it again.”

In one example, Lou discovered that his definition of “relevant” wasn’t the same as the LLM’s. Seeing the explanation helped him identify the mismatch, adjust the language in the prompt, and test it again.

“It just accelerated our eval development and our prompt engineering revision extremely fast,” he says.

Instead of treating an incorrect evaluation as a dead end, the explanation becomes another signal the team can use to improve both its prompts and its evaluation framework.

Building an evaluation loop that gets better with production

As Bazaarvoice’s AI systems move toward production, the team’s evaluation datasets can continue evolving with them.

Real-world failures can become new test cases. Those examples can be added back into evaluation datasets, helping the team test future versions against problems that actually occurred.

Over time, this creates a feedback loop between what happens in production and what the team tests next.

For Lou, the lesson is less about finding the perfect evaluation framework from day one and more about building one that can improve alongside the AI system.

Start with a clear question. Create a small set of trusted human labels. Benchmark the evaluator against them. Then use that evaluator to understand quality across much larger datasets.

The result is a workflow that lets Bazaarvoice spend less time manually labeling AI outputs and more time understanding and improving the experiences it is building.