A shadow deployment runs a candidate model or agent version against live production traffic while the existing version keeps serving users. Real requests are mirrored to the candidate, its outputs are logged and compared against the incumbent, and nothing it produces is ever returned to a user. Also called shadow mode or shadow testing, it shows how a new version behaves on the real input distribution before anyone depends on it.
It exists because offline evaluation on a static dataset does not surface behavior that only appears under real traffic. Your eval set was assembled by people who already know what the system is supposed to do. Production contains the malformed inputs, the languages you did not plan for, the 4,000-token pasted document, and the user who asks three questions in one message. A candidate can score well on your dataset and still behave differently on half of a Tuesday’s traffic.
Key takeaways
- A shadow deployment sends real traffic to a candidate version and discards its output. Users are never exposed, so the blast radius of a bad candidate is zero.
- It answers questions about behavior, latency, cost, and error rate on the real input distribution. It cannot answer questions about user response, because no user ever sees the output.
- Shadow, canary, and A/B are three different tools. Shadow means no exposure and no outcome data; canary means limited exposure with real outcome data; A/B means deliberate split traffic for statistical comparison.
- For agents, shadowing is harder than for models. Any tool call that writes data, sends a message, or spends money has to be disabled or sandboxed, and multi-turn runs diverge from reality after the first differing response.
- Comparing outputs without ground truth is the hard part, and it is where reference-free evals and pairwise judging do the work a label would otherwise do.
How it works
The mechanics are the same whether you are shadowing an XGBoost model or a multi-step agent:
- Mirror the request. A proxy, a service-level fork, or a background task copies the incoming request and sends it to the candidate as well as the incumbent.
- Serve only the incumbent. The user’s response path is untouched, and the candidate call is asynchronous so it cannot add latency to a request someone is waiting on.
- Log both outputs with a shared correlation ID, along with latency, token counts, errors, and whatever intermediate steps the candidate took.
- Compare offline. Agreement rate, output distribution, eval score distribution, cost per request, tail latency, and error rate.
- Promote or discard. A shadow run that looks good becomes a canary. One that surfaces a problem becomes an eval case you keep forever.
Mirroring is a deployment-layer concern more than a modeling one, which is why it usually lands in the serving stack rather than in model code. The comparison side depends on the same production telemetry that monitoring a live model already requires, which is why the two usually get built together.
Shadow deployment versus canary and A/B testing
Shadow deployment. Candidate sees 100% of traffic, or a sample of it, and 0% of users see its output. You learn how it behaves. You learn nothing about whether people like it, because nobody experienced it. Risk to users is zero; cost is real, since you pay for inference twice.
Canary release. A small share of users, often 1% to 5%, gets the candidate’s actual output. You now have outcome data: clicks, resolutions, escalations, complaints. You have also accepted real exposure, so you need automatic rollback triggers on your guardrail metrics.
A/B test. Traffic is deliberately split, usually 50/50 or some planned allocation, and held long enough to reach statistical significance on a chosen metric. The goal is a defensible causal comparison, not a safety check.
They form a sequence, not a menu. Shadow first, to confirm the candidate does not crash, hallucinate more, cost triple, or time out on long inputs. Canary next, to see what happens when a real person receives the output. A/B last, to prove which version is better on a business metric.
What shadow mode cannot tell you
No outcome signal. There is no click, no thumbs up, no resolved ticket. Every conclusion has to come from evaluating the output itself.
Conversations diverge immediately. In a multi-turn agent, the second user message is a response to what the incumbent said. The shadow version’s first reply was never seen, so from turn two onward you are feeding it a conversation that would not have existed if it had been live. Shadow results for single-turn behavior are trustworthy. Shadow results for a ten-turn session are directionally useful at best.
Side effects are dangerous. A model produces a prediction. An agent calls tools. If the candidate can issue a refund, send an email, or write to a database, shadow mode becomes a production incident. Tools have to be stubbed, sandboxed, or restricted to read-only, and that changes the behavior you are measuring.
Cost and rate limits are real. Every shadowed request is a second inference call. Sampling a slice of traffic usually gives you enough signal at a fraction of the spend. How thin a slice you can get away with depends on your volume and on how rare the behavior you are hunting is, so size the sample against the rate of the thing you are trying to catch, not against a round percentage.
Comparing outputs when there is no correct answer
For a classifier, comparison is easy: measure agreement, then wait for labels and compare accuracy. For generative systems, two different strings can both be right, so agreement rate is a weak signal on its own.
What works instead is scoring both outputs with the same evaluators. Code checks for anything deterministic: schema validity, required fields, forbidden content, length, tool-call correctness. A judge model for the semantic parts, using the setup in the guide to LLM as a judge. Pairwise judging, where the judge sees both outputs and picks the better one, is often more reliable than scoring each independently, because relative comparison is an easier task.
All of this requires that both versions emit comparable traces, which is the argument for instrumenting first and shadowing second. The structure is covered in AI agent tracing and evaluation.
FAQ
What is the difference between shadow deployment and canary deployment?
Exposure. In a shadow deployment the candidate’s output is discarded and no user sees it. In a canary release a small percentage of real users receive the candidate’s output. Shadow is safer and tells you less; canary is riskier and tells you what users do.
How long should a shadow deployment run?
Long enough to cover the traffic patterns you care about, which usually means at least one full weekly cycle. Weekday and weekend traffic differ, and so do business hours and overnight. If you have known seasonal spikes, wait for one or replay a previous one.
Does shadow deployment work for LLM agents?
Partly. It works well for single-turn behavior, latency, cost, and tool selection, as long as write-capable tools are disabled. It works poorly for long sessions, because the shadow conversation diverges from the real one after the first differing response. Shadow the single-turn parts, and rely on canary exposure for anything that depends on a full session.
What should I measure during a shadow run?
Error and timeout rate, p50 and p99 latency, cost or token count per request, output distribution compared to the incumbent, eval scores broken out by slice, and the rate of disagreement on cases where the incumbent was known to be correct. Disagreements are the most valuable output: each one is a candidate test case.
Is shadow deployment worth the doubled inference cost?
For a high-risk change, usually yes, and you can sample rather than mirror everything. For a low-risk change with strong offline coverage and a fast rollback path, a canary with automatic rollback is the better trade. The case for continuous production visibility is made in what an agent observability platform does.
