Shadow testing runs a new model, prompt, retriever, or agent version alongside production on the same inputs, without showing its output to anyone. Production answers the user. The shadow version answers into a log. You then compare the two responses, or score both with an evaluator, and decide whether the candidate is ready to be seen.
The appeal is real production inputs at zero user risk. Offline eval sets are curated and they age; live traffic has the malformed pastes, the multilingual inputs, the 40-turn conversations, and the queries nobody wrote a test case for.
The constraint is the other half of the same fact. No user sees the output, so no user reacts to it. No click, no retry, no escalation, no thumbs-down. You have real inputs and no outcome signal, so every conclusion comes from comparing the shadow output to the production output or from scoring it.
Build better agents with Arize
Trace, evaluate, and learn. Build agents that work with Arize AX and start tracing your runs today.
Prefer open source? Try Arize Phoenix for self-hosted, open source agent observability.
Key takeaways
- Shadow testing gives you real inputs and no user reaction, so the only signals available are output comparison against the current version and evaluator scores.
- A shadowed agent that executes real tool calls is not shadow testing. It is a second production system. Every write, payment, email, and ticket has to be stubbed or sandboxed.
- Budget for roughly double inference spend on shadowed traffic plus scoring on top, which is why most teams shadow a stratified sample rather than everything.
- Run the shadow call asynchronously and off the user’s request path so a slow or failing candidate cannot degrade production latency.
- Outputs differ even between identical versions, so measure a baseline diff rate before reading a candidate’s diff rate as a signal.
What you can actually learn
Behavioral and structural differences. Which tools the candidate calls and in what order, how many steps it takes, whether it asks a clarifying question where production answered directly, how often it refuses. These are countable without any judgment about quality, and they are usually where a surprising regression shows up first.
Cost and latency under real load. A candidate that looks fine on an eval set can be slower on the long-context tail of production traffic, or can quietly add tool calls that change cost per request.
Failure and error rates. Malformed tool arguments, schema violations, timeouts, parse failures, and context-length overruns surface on the exact inputs that trigger them.
Relative quality, with an instrument. For a quality verdict you score both outputs. Either a judge grades each response independently against a rubric, or it compares the two pairwise on the same input. Pairwise tends to be more stable, but it needs position randomization because judges favor whichever response they read first.
Two mechanics matter before any of this is trustworthy. Run the shadow call asynchronously, off the user’s request path, so a slow candidate cannot add latency to the real response. And measure production’s diff rate against itself first, because sampling alone makes two runs of one version disagree, and a candidate’s diff rate means nothing without that floor.
What you cannot learn is whether users prefer the new version. That requires serving the output, which is what a canary or an A/B test is for.
Side effects are the whole problem for agents
An LLM that only generates text is straightforward to shadow. An agent is not, because an agent acts. If the shadow run issues the refund, sends the email, or updates the CRM record, you have not built a test environment. You have built a second production system nobody is watching, with the same authority and none of the oversight.
Three ways teams handle this, in increasing order of fidelity and effort:
- Replay recorded responses. Tool calls are intercepted and answered with what the production run received. Cheap and deterministic, but the moment the candidate calls a tool production did not, or calls it with different arguments, there is no recorded response and the trajectory diverges into guesswork.
- Route to read-only or sandbox implementations. Reads hit real systems so retrieval and lookups stay realistic; writes go to a sandbox or a no-op that records the intended call. Usually the right tradeoff, and it depends on the tool layer being configurable per environment rather than hardcoded, which is one of the controls harness engineering exists to provide.
- Full sandbox environment. Highest fidelity, highest maintenance, and it drifts from production unless someone owns keeping it current.
Whichever you pick, the suppressed actions are data. A candidate that would have issued far more refunds than production is telling you something, and the record of blocked writes is often the most valuable artifact of the run. This is the gap that traditional software testing leaves open for agents: the code is correct and the decision is wrong.
What it costs
Shadow testing runs each shadowed request twice, so inference spend on that slice roughly doubles, and scoring both outputs with a judge adds a third stream of inference. That is the honest reason most teams shadow a sample, and it is worth working out where evaluation cost accumulates before pointing a judge at every shadowed pair.
Sample deliberately rather than randomly. A stratified sample that oversamples the intents, languages, and long-conversation cases you are least sure about teaches you more per dollar than uniform sampling, which spends its budget confirming the easy majority still works.
Two costs are easy to miss. The comparison needs somewhere to live, meaning paired traces, a diff view, and score storage. And shadow traffic touches real user data in a second code path, so retention, redaction, and access controls apply exactly as in production.
Shadow testing, canaries, and A/B tests
The distinction is who sees the output. Shadow: nobody, so risk is zero and user signal is zero. Canary: a small share of real users, so risk is real but bounded and that slice produces live outcome signal. A/B test: both versions serve users in a split sized for a statistical comparison.
They are a sequence more often than alternatives. Shadow catches crashes, cost blowups, and behavioral divergence at no user risk. The canary confirms real users tolerate the output. An A/B test comes last, when the question is which version is better rather than whether the new one is safe.
FAQ
What is the difference between shadow testing and shadow evaluation?
They usually describe the same practice. Shadow testing emphasizes the traffic mechanism, running the candidate on live inputs without serving it. Shadow evaluation emphasizes the scoring applied to the outputs that mechanism produces.
Is shadow testing the same as detecting unsanctioned LLM usage?
No, and the vocabulary collides. Detecting unapproved model or tool usage inside an organization is a governance concern, sometimes called shadow AI. Shadow testing is a release practice for a version you own and intend to ship. Shadow means hidden from users in one case and hidden from IT in the other.
How do I compare shadow outputs when there is no correct answer?
Two options answering different questions. Pairwise preference asks a judge which of the two responses is better for this input, which is the right framing for a ship decision. Independent rubric scoring grades each response against criteria such as faithfulness to retrieved context or policy adherence, which tells you whether either version is good in absolute terms. Use pairwise for the decision and rubric scoring for the diagnosis.
Can shadow testing replace offline evals?
No. Offline evals run against known-correct answers on a fixed dataset, which is what makes a regression check repeatable. Shadow testing has real inputs and no labels. Offline evals catch regressions on cases you already decided are correct; shadow testing catches behavior on inputs you never imagined.
