A/B testing for LLMs splits live traffic between two or more versions of an AI system and compares what happens on real user outcomes. The variants might differ by model, prompt, retrieval strategy, tool policy, or agent workflow. Everything else about the request path stays the same, so the difference in outcomes is attributable to the one thing you changed.
The mechanics are borrowed from product experimentation and they transfer cleanly. What does not transfer is the outcome metric. A checkout button has a conversion event; a support agent’s answer has a quality nobody logged. So the first real decision is not the traffic split, it is what counts as the outcome, and both available answers cost you something. You can use a behavioral proxy users generate for free, or score outputs with an evaluator you have to build and trust.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Both arms are served to users. That is what separates an A/B test from shadow testing and from a canary: the purpose is comparison, not risk containment.
- The outcome metric is the hard part. Pick a behavioral proxy the product already emits or an evaluator score, and know the failure mode of the one you chose.
- Significance is harder than in classic A/B testing because judge variance stacks on user variance, so the same effect size needs more traffic to detect.
- Assign at the session or account level and keep it sticky. Per-request assignment mixes both variants inside one conversation.
- Carry latency and cost per resolved task as guardrail metrics. A variant that wins on quality and doubles token spend is a decision, not a victory.
Choosing what to measure
Behavioral proxies are signals the product already produces: whether the user retried, rephrased, escalated to a human, abandoned the session, or came back the next day. They are cheap, unbiased by any model you own, and they reflect what people did rather than what a scorer thinks. The weakness is coverage. Most sessions emit no strong signal, and thumbs-up rates are dominated by the small, unrepresentative group who bother to click.
Evaluator scores fill that gap by scoring every response, or a sample, with a judge model or a deterministic check. Now every session contributes a number. The cost is a measurement instrument sitting in the middle of the comparison with biases of its own. A judge built on the same model family as one variant can prefer that variant’s phrasing for reasons unrelated to quality, which is the practical reason to account for self-evaluation bias when a model grades its own outputs before letting a judge declare a winner.
Most teams run both. The proxy is the metric of record because it is closer to the business outcome. The evaluator score explains a proxy movement and catches regressions the proxy is too sparse to see.
Running the split
Decide the assignment unit first. Request-level assignment gives the most samples and the least valid experiment for anything conversational, since a user can get variant A’s answer and variant B’s follow-up in one thread. Session or user assignment costs power and buys a coherent experience to measure.
Then instrument so every span carries the variant identifier along with the prompt version, model version, and tool configuration in effect. That is the difference between knowing variant B won and knowing why. When B’s win comes entirely from one intent category, it lives in the traces and not in the aggregate, which is the standing argument for tracing and evaluating agents at the span level rather than logging one score per request. For guidance on testing agent behavior, see agent evaluation.
Guardrail metrics run alongside the primary one without the same patience: p95 latency, cost per resolved task, tool error rate, and safety violations should each be able to stop an experiment on their own.
Why significance is harder here
Classic A/B testing measures a binary event with known variance. An LLM A/B test usually measures a continuous score from an instrument that does not return the same value twice on identical input. That variance sits on top of the natural variation between users and widens the confidence interval around any difference you observe.
Three habits help. Measure the judge’s run-to-run spread on a fixed set before launch, so you know how much of an observed difference could be the scorer moving. Use paired comparison where you can afford to generate both outputs, since scoring which of two responses to the same input is better removes between-user variance entirely. And fix the analysis plan before launch, because checking a dashboard daily and stopping when it looks good will find a winner in an experiment with no effect.
Agent variants add a wrinkle. Two architectures can reach the same answer quality through different step counts, tool mixes, and failure modes, so a single win rate hides the tradeoff you care about. Comparing trajectories rather than final answers alone is one of the failures that traditional software testing misses when applied to agents.
How it relates to shadow testing and canaries
All three put a new version in front of production traffic and differ in who sees the output. In an A/B test both arms are served and the point is a comparison sized to detect a difference. A canary serves the new version to a small share of users and watches for damage before widening, which makes it a rollout strategy with a bounded blast radius rather than an experiment. Shadow testing serves the output to nobody, removing the risk along with any user reaction to measure. They stack: shadow first, canary for safety, then an A/B test when the question is which version is better.
FAQ
How do I A/B test prompts in an LLM app?
Treat each prompt as a versioned artifact with an ID, route traffic between versions at the session level, and record the version ID on every span the request produces. Then compare a behavioral proxy such as retry or escalation rate alongside an evaluator score on the same sessions. The part teams skip is the version ID on the span. Without it you cannot reconstruct which prompt produced which trace once the experiment ends.
Can I use an LLM judge to decide the winner?
Yes, with two conditions. Validate the judge against human labels on a sample of this specific traffic, because a judge that agrees with reviewers on a benchmark can disagree badly on your data. And avoid a judge from the same model family as one of the variants, or use a jury of several judges, so stylistic self-preference does not decide the outcome.
How do I compare agent architectures and compute win rates automatically?
Run the architectures as variants behind the same entry point, score each completed session with the same evaluator suite, and compute the win rate per intent category rather than overall. Pair that with trajectory metrics: steps per task, tool call distribution, and cost per resolution. An architecture that wins on answer quality while tripling tool calls is not the one to ship, and only the trajectory view surfaces that.
How long should an LLM A/B test run?
Long enough to cover a full traffic cycle including any weekly usage pattern, and long enough for the arm sizes to detect the effect you care about given your measured score variance. Compute that sample size before launch. Stopping the moment a dashboard shows a lead is the most common way teams ship a variant that was never better. If neither variant wins, check the slices before filing the result, since a change that helps one intent category and hurts another nets to zero in the aggregate while being real in both directions.
