What Is Shadow Deployment?
Learn how shadow deployment mirrors real traffic to a new version with zero user risk, and how it compares to canary releases.
Expected Interview Answer
Shadow deployment, also called traffic mirroring or dark launching, is a release strategy where real production traffic is duplicated and sent to a new version running silently alongside the live version, letting the new version process genuine requests and be observed for correctness and performance without its responses ever being returned to actual users.
A proxy, service mesh, or load balancer copies each incoming live request and sends one copy to the stable version, which serves the real response to the user, while an identical copy is sent asynchronously to the shadow (new) version, whose response is discarded or only logged for comparison. Because the shadow version’s output never reaches real users, there is zero user-facing risk even if it crashes, returns wrong data, or is dramatically slower — engineers compare the shadow’s outputs, latency, and error rates against the stable version’s to validate correctness before ever exposing real users to it. Shadow deployment is especially valuable for testing changes with side effects that are hard to simulate in staging, such as machine learning model updates or a rewritten backend service, but it requires care to avoid duplicate side effects like double-charging a payment or double-sending an email, typically solved by routing writes to a sandboxed dependency or stubbing external calls in the shadow path. Unlike canary, which risks real users on a small scale, shadow deployment risks no real users at all, but it also cannot validate real user-facing behavior like UI rendering, since responses are never actually shown.
- Zero user-facing risk since shadow responses are never returned to real users
- Validates new code against genuine production traffic volume and patterns
- Surfaces performance regressions before any real exposure
- Especially effective for high-risk changes like ML model updates or rewritten services
AI Mentor Explanation
Shadow deployment is like having a trainee umpire silently make every decision alongside the real umpire during a live match, without any of the trainees calls ever being shown on the scoreboard or affecting the game. The real umpires decisions run the match exactly as normal, while the trainees decisions are recorded privately and compared afterward for accuracy. If the trainee gets several calls wrong, nobody in the stadium ever knew, and the real match was never affected. Only once the trainees accuracy is proven does the board consider using them in an actual official capacity.
Step-by-Step Explanation
Step 1
Deploy the shadow version
Run the new version alongside stable, receiving mirrored traffic but never serving real responses.
Step 2
Mirror live requests
A proxy or service mesh duplicates each incoming request, sending one copy to stable and one to shadow.
Step 3
Discard shadow responses
The shadow version processes the request fully but its response is discarded or only logged, never returned to the user.
Step 4
Compare and promote
Engineers compare shadow output, latency, and errors against stable; once validated, the new version can move to canary or full rollout.
What Interviewer Expects
- Understanding that shadow responses are never returned to real users
- Awareness of how mirrored/duplicated traffic is generated (proxy or service mesh)
- Knowledge of the side-effect risk (e.g. duplicate payments/emails) and how to mitigate it
- Ability to explain when shadow deployment is preferred over canary or blue-green
Common Mistakes
- Confusing shadow deployment with canary, which does expose real users to the new version
- Forgetting to stub or sandbox side-effecting calls, risking duplicate real-world actions
- Assuming shadow deployment validates UI/UX, when it can only validate backend behavior
- Not accounting for the extra infrastructure and compute cost of processing every request twice
Best Answer (HR Friendly)
“Shadow deployment means we send a real copy of live traffic to the new version silently in the background, without ever showing its response to an actual user — the current version keeps handling everyone as normal. We compare the shadow versions behavior, speed, and accuracy against the live version, so we can catch problems on genuine production traffic with zero risk to real users, before we ever consider rolling it out for real.”
Code Example
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: myapp
spec:
hosts:
- myapp.example.com
http:
- route:
- destination:
host: myapp
subset: stable
weight: 100
mirror:
host: myapp
subset: shadow
mirrorPercentage:
value: 100.0
# Real users only ever see the “stable” response; "shadow" output is discardedFollow-up Questions
- How would you prevent duplicate side effects, like double-charging a payment, in a shadow deployment?
- What kinds of changes are best validated with shadow deployment versus canary?
- How would you compare shadow-version output against stable-version output at scale?
- What extra infrastructure cost does shadow deployment introduce?
MCQ Practice
1. What happens to the shadow versions response in a shadow deployment?
The defining trait of shadow deployment is that the shadow versions output never reaches real users — it is only observed.
2. What is a key risk that must be mitigated in shadow deployment?
Since requests are mirrored to the shadow version, any side-effecting calls it makes can duplicate real-world actions unless sandboxed or stubbed.
3. Why is shadow deployment often preferred over canary for testing an ML model update?
Shadow deployment lets teams validate a risky change like a model update against genuine traffic without any user-facing exposure.
Flash Cards
What is shadow deployment? — Mirroring real traffic to a new version whose responses are never shown to real users.
How is traffic mirrored? — A proxy or service mesh duplicates each request, sending copies to both stable and shadow.
Main risk of shadow deployment? — Duplicate side effects (e.g. payments, emails) unless the shadow path sandboxes them.
What can shadow deployment NOT validate? — Real user-facing behavior like UI rendering, since responses are never actually shown.