Canary evaluation routes a small share of real traffic, or a limited set of users, through a new AI system version and evaluates the result before rolling out further. It is for the cases offline evals cannot predict, which is most of them: your eval set does not contain next Tuesday’s inputs.
The defining property is that canary users see the output. That is the whole tradeoff. Unlike shadow testing, where the candidate’s response goes to a log, a canary exposes real people to a version you are not sure about, and in exchange you get the one thing shadow testing cannot produce: how users respond. The risk is real and bounded, and the evaluation exists to catch a bad version while it is still affecting a small group.
A canary without evaluation is just a risky rollout with a smaller blast radius. The traffic shift is the delivery mechanism; the scoring, guardrails, and predefined rollback criteria are what make it a test.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Canary users receive the new version’s output. That is what separates a canary from shadow testing, and it makes rollback criteria mandatory rather than optional.
- Infrastructure health checks pass while an AI system produces wrong answers, so a canary needs semantic signals and not just error rate, latency, and saturation.
- The core tension is blast radius against statistical power. A smaller share is safer and takes proportionally longer to tell you anything.
- Low-traffic systems often cannot run a meaningful canary. Route by cohort, extend the bake, or use shadow testing plus a staged internal rollout instead.
- Split signals by speed: guardrail violations should trigger rollback on a single occurrence, while quality scores need volume before they mean anything.
What makes an AI canary different
A conventional canary deploy watches the golden signals: error rate, latency, saturation, and traffic. An AI version can hold all of them steady while giving worse answers. HTTP 200 responses, normal p95 latency, no exceptions, and a model that started hedging on the questions your users care about most.
So the canary carries semantic signals alongside the operational ones, arriving on different timescales.
Guardrail violations, immediately. Policy breaches, PII leakage, refusals on legitimate requests, schema violations, unauthorized tool calls. Each should stop the rollout on its own without waiting for a statistical argument, because one occurrence is already the event you were trying to prevent.
Behavioral counters, within minutes to hours. Steps per session, tool call distribution, retry and rephrase rate, escalation, abandonment, cost per resolved task. These need no labels, accumulate quickly, and usually move before quality scores do.
Quality scores, slowest. Online evaluation runs where no reference answer exists, so it relies on reference-free scorers: faithfulness to retrieved context, policy adherence, task completion, answer relevance. Deciding where an LLM judge is the right instrument matters more here than offline, because you pay to score live traffic and act on the result while users are still in the canary.
All three need the candidate version identified on the traces they produce. If the canary’s spans do not carry the version, model, and prompt identifiers, you can see that something got worse without being able to attribute it, which is why tracing and evaluation on live agent traffic comes before progressive delivery rather than during it.
Sizing the canary
Two forces pull in opposite directions. A smaller share limits how many users a bad version reaches. A larger share reaches a conclusion sooner. There is no universally correct number, and starting with a single-digit percentage of traffic and widening in stages is a convention inherited from conventional deploys, not a figure derived from your data.
What should drive the decision is how long it takes to accumulate enough observations of your slowest signal. If escalation rate is the metric of record and escalations are rare, a tiny share takes days to produce a readable difference, and a canary that runs for days gets forgotten or overridden by the next release. Work backward: decide the effect size worth catching, estimate the sessions that needs given your measured variance, then pick the share and bake time that produce them in a window your team will wait through. Cover a full usage cycle while you are at it, including at least one low traffic period, since a canary that runs for an hour at peak has seen one kind of user.
If that calculation says weeks, the honest conclusion is that your traffic cannot support a statistically meaningful canary. That is common for internal tools and early-stage products. Run shadow testing for divergence and cost, roll out by cohort starting with employees, and rely on guardrail violations rather than aggregate quality movement as the stop condition.
Running one
Choose the routing unit. Assign by user or session, not by request, and make it sticky. A user who gets the new version mid-conversation experiences a personality change, and the session becomes uninterpretable for both.
Choose the cohort deliberately. Random sampling gives representative results; starting with internal users gives forgiving results and a safer first hour. Many teams do both in sequence. Avoid a cohort picked for convenience that happens to be your largest accounts.
Write the rollback criteria before launch. Which signals stop the rollout, at what level, and whether rollback is automatic or paged to a human. Criteria written afterward get negotiated against the deadline.
Define the widening schedule and bake time per stage. A stage that widens the moment the dashboard looks fine is not a canary, because the failures worth catching need volume or elapsed time to appear.
Make rollback fast. Route back by configuration rather than redeploying. Prompt and model versions should be swappable at runtime, and any state the new version wrote, such as memory records or altered tickets, needs a plan up front. Keeping the operational and trace-level pictures in one place is much of what an agent observability platform is for during a rollout.
FAQ
What is a canary in AI deployment?
The same idea as a canary release in conventional software: send a small fraction of live traffic to the new version and watch before sending the rest. The AI-specific part is what you watch. Operational health is not sufficient, because a model can serve fast, error-free, incorrect answers all day, so canary evaluation adds output scoring and behavioral comparison against the version still serving everyone else.
How is a canary different from shadow testing and from A/B testing?
Who sees the output. Shadow testing serves the candidate’s output to nobody, so risk is zero and there is no user reaction to measure. A canary serves it to a small group, accepting bounded risk in exchange for real user signal. An A/B test serves both versions at sizes chosen for a statistical comparison, so its purpose is deciding which is better rather than whether the new one is safe to widen.
What should trigger an automatic rollback?
Anything you would not debate at 2am: safety or policy violations, PII exposure, a spike in tool errors or timeouts, latency past what users notice, cost per request outside the budget. Quality score movements are better routed to a human, because judge variance produces false alarms and a rollback triggered by scorer noise teaches the team to disable the automation.
Can I run a canary without enough traffic to be statistically valid?
Yes, as long as you are clear about what it buys you. At low volume a canary catches crashes, obvious behavioral breakage, and guardrail violations, which are the failures that hurt most anyway. It will not confirm a small quality improvement. For that you need more traffic or a paired offline comparison on a fixed dataset.
