Canary Deployment Cheat Sheet
Techniques for gradually shifting production traffic to a new version while monitoring metrics to catch regressions early.
2 PagesAdvancedFeb 10, 2026
Core Concept
How canary releases limit blast radius.
- Canary- A small subset of instances running the new version, receiving a small percentage of traffic
- Baseline- The stable version still serving the majority of traffic during the canary phase
- Progressive rollout- Gradually increasing the canary's traffic share (e.g. 5% -> 25% -> 50% -> 100%) as confidence grows
- Automated analysis- Comparing canary vs baseline metrics (error rate, latency) to auto-promote or auto-rollback
- Blast radius- The scope of users affected if the new version has a defect; canary keeps it small initially
- Feature flags- Often paired with canaries to decouple deployment from feature exposure
Nginx Weighted Canary
Split traffic between stable and canary upstreams by weight.
nginx
upstream backend { server 10.0.0.1:3000 weight=9; # stable, ~90% server 10.0.0.2:3000 weight=1; # canary, ~10%}server { listen 80; location / { proxy_pass http://backend; }}
Kubernetes Canary (replica ratio)
Approximate traffic split via replica counts behind one Service.
yaml
# stable: 9 replicas, canary: 1 replica -> Service load-balances ~90/10apiVersion: apps/v1kind: Deploymentmetadata: name: myapp-stablespec: replicas: 9 selector: matchLabels: {app: myapp} template: metadata: labels: {app: myapp, track: stable}---apiVersion: apps/v1kind: Deploymentmetadata: name: myapp-canaryspec: replicas: 1 selector: matchLabels: {app: myapp} template: metadata: labels: {app: myapp, track: canary}
Metrics to Watch
What to monitor before promoting a canary.
- Error rate- HTTP 5xx rate on canary compared to baseline over the same window
- Latency (p95/p99)- Tail latency regressions often surface before average latency does
- Saturation- CPU/memory/connection pool usage on canary instances
- Business metrics- Conversion rate, checkout success, etc., for user-facing regressions that infra metrics miss
Argo Rollouts: Analysis-Gated Canary
Define progressive traffic steps that pause for automated metric analysis before continuing.
yaml
apiVersion: argoproj.io/v1alpha1kind: Rolloutmetadata: name: myappspec: replicas: 10 strategy: canary: steps: - setWeight: 10 - pause: {duration: 5m} - analysis: templates: - templateName: success-rate - setWeight: 50 - pause: {duration: 10m} - setWeight: 100---apiVersion: argoproj.io/v1alpha1kind: AnalysisTemplatemetadata: name: success-ratespec: metrics: - name: success-rate interval: 1m successCondition: result[0] >= 0.95 failureLimit: 3 provider: prometheus: address: http://prometheus.monitoring:9090 query: | sum(rate(http_requests_total{app="myapp",status!~"5.."}[5m])) / sum(rate(http_requests_total{app="myapp"}[5m]))
Flagger Canary CRD
Declarative canary with built-in metric thresholds and auto-rollback, driven off a Kubernetes HPA target.
yaml
apiVersion: flagger.app/v1beta1kind: Canarymetadata: name: myappspec: targetRef: apiVersion: apps/v1 kind: Deployment name: myapp service: port: 80 analysis: interval: 1m threshold: 5 # max consecutive failed checks before rollback maxWeight: 50 stepWeight: 5 # increase canary traffic by 5% each interval metrics: - name: request-success-rate thresholdRange: {min: 99} interval: 1m - name: request-duration thresholdRange: {max: 500} interval: 1m
Traffic Mirroring (Shadow Traffic)
Send a copy of live requests to the canary without it ever affecting real responses — zero user-facing risk.
yaml
apiVersion: networking.istio.io/v1beta1kind: VirtualServicemetadata: name: myappspec: hosts: - myapp http: - route: - destination: {host: myapp, subset: stable} weight: 100 mirror: host: myapp subset: canary mirrorPercentage: value: 100.0 # canary sees 100% of real traffic but its responses are discarded
PromQL Canary Judge Query
Compare canary vs. stable error rate directly, rather than eyeballing two dashboards.
promql
# Error rate delta between canary and stable tracks (positive = canary is worse)( sum(rate(http_requests_total{app="myapp",track="canary",status=~"5.."}[5m])) / sum(rate(http_requests_total{app="myapp",track="canary"}[5m])))-( sum(rate(http_requests_total{app="myapp",track="stable",status=~"5.."}[5m])) / sum(rate(http_requests_total{app="myapp",track="stable"}[5m])))> 0.02 # alert/rollback if canary error rate is >2 percentage points worse than stable
Canary Pitfalls
Ways a canary rollout gives false confidence.
- Non-representative traffic- A 5% slice may miss the one enterprise customer or region that triggers the bug ('canary washing')
- Sticky/session-affine routing- If the same users always land on canary, you're really doing a small permanent rollout, not a randomized sample
- Insufficient sample size- Low-traffic services may not generate enough requests in the analysis window to reach statistical significance
- Metric lag vs. promotion speed- If steps auto-advance faster than metrics/logs are ingested, a regression can be masked until it's too late
- Shared caches/DBs contaminating baseline- A buggy canary write can corrupt shared state that the stable track then also reads, invalidating the comparison
- Downstream blast radius- A canary calling a shared downstream (e.g. exhausting a connection pool or rate limit) can degrade the STABLE track too
- Vanity metrics only- Watching only infra metrics (CPU, latency) while missing business metrics (conversion, error toasts) that users actually feel
Pro Tip
Automate the rollback decision based on statistically significant metric deltas (not just eyeballing a dashboard) — tools like Flagger or Argo Rollouts can halt and revert a canary the moment error rates spike, faster than any human on-call.
Was this cheat sheet helpful?
Explore Topics
#CanaryDeployment#CanaryDeploymentCheatSheet#DevOps#Advanced#CoreConcept#NginxWeightedCanary#Kubernetes#Canary#ErrorHandling#CheatSheet#SkillVeris
