Blue-Green Deployment Cheat Sheet
Patterns for running two identical production environments to enable zero-downtime releases and instant rollback.
Core Concept
How blue-green deployments work at a high level.
- Blue environment- The currently live production environment serving all user traffic
- Green environment- An idle, identical environment where the new version is deployed and validated
- Traffic switch- A router/load-balancer/DNS change that instantly cuts traffic over from blue to green
- Rollback- Simply switch traffic back to blue if green shows errors post-cutover
- Smoke tests- Automated checks run against green before it receives real traffic
- Database compatibility- Schema changes must be backward-compatible so both blue and green can operate against the same DB during transition
Kubernetes Example
Switch a Service's selector to cut over traffic.
# Service currently points at the 'blue' DeploymentapiVersion: v1kind: Servicemetadata: name: myappspec: selector: app: myapp version: blue # change to 'green' to cut over ports: - port: 80 targetPort: 8080
Cutover & Rollback
Scripted traffic switch and rollback commands.
# Deploy green alongside blue, then verify it's healthykubectl apply -f green-deployment.yamlkubectl rollout status deployment/myapp-green# Cut traffic overkubectl patch service myapp -p '{"spec":{"selector":{"version":"green"}}}'# If green is unhealthy, roll back instantlykubectl patch service myapp -p '{"spec":{"selector":{"version":"blue"}}}'
Expand/Contract DB Migrations
Keep the schema backward-compatible so blue and green can share one database during cutover.
-- PHASE 1: EXPAND (deploy before green goes live)-- Add the new column as nullable; old (blue) code ignores it, new (green) code writes itALTER TABLE orders ADD COLUMN shipping_status text NULL;-- Backfill in batches so it doesn't lock the tableUPDATE orders SET shipping_status = 'unknown' WHERE shipping_status IS NULL AND id BETWEEN 1 AND 100000;-- PHASE 2: cut traffic blue -> green (both still read/write compatible schema)-- PHASE 3: CONTRACT (only after blue is fully decommissioned, no rollback path needed)ALTER TABLE orders ALTER COLUMN shipping_status SET NOT NULL;ALTER TABLE orders DROP COLUMN legacy_status;
DNS-Based Cutover (Route 53)
Shift traffic at the DNS layer using weighted records tied to health checks.
# Start: blue=255, green=0 (all traffic on blue)aws route53 change-resource-record-sets --hosted-zone-id Z123 \ --change-batch '{ "Changes": [{ "Action": "UPSERT", "ResourceRecordSet": { "Name": "app.example.com", "Type": "A", "SetIdentifier": "green", "Weight": 255, "AliasTarget": {"HostedZoneId": "Z2ABC", "DNSName": "green-lb.example.com", "EvaluateTargetHealth": true} } }] }'# Then drop blue's weight to 0 once green's health check is passingaws route53 change-resource-record-sets --hosted-zone-id Z123 \ --change-batch '{"Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"app.example.com","Type":"A","SetIdentifier":"blue","Weight":0,"AliasTarget":{"HostedZoneId":"Z2ABC","DNSName":"blue-lb.example.com","EvaluateTargetHealth":true}}}]}'# NOTE: DNS TTL + client-side caching means this cutover is NOT instant like a k8s# selector patch -- expect stragglers on blue for minutes to hours depending on TTL.
Istio VirtualService Cutover
Use a service mesh route rule for an atomic, mesh-wide traffic switch with no DNS propagation delay.
apiVersion: networking.istio.io/v1beta1kind: VirtualServicemetadata: name: myappspec: hosts: - myapp.internal http: - route: - destination: host: myapp subset: blue weight: 0 # flip to 100 to cut fully to blue - destination: host: myapp subset: green weight: 100 # flip to 0 to roll back instantly---apiVersion: networking.istio.io/v1beta1kind: DestinationRulemetadata: name: myappspec: host: myapp subsets: - name: blue labels: {version: blue} - name: green labels: {version: green}
Graceful Connection Draining
Drain in-flight requests from the outgoing color before it stops receiving new connections entirely.
# 1. Remove blue from the load balancer target group (stops NEW connections)aws elbv2 deregister-targets --target-group-arn $BLUE_TG --targets Id=$INSTANCE_ID# 2. Poll until connection draining completes (default deregistration_delay = 300s)aws elbv2 describe-target-health --target-group-arn $BLUE_TG \ --query 'TargetHealthDescriptions[?Target.Id==`'"$INSTANCE_ID"'`].TargetHealth.State'# 3. Only once state == 'unused' is it safe to terminate the blue instance/podkubectl delete deployment myapp-blue --grace-period=300
Blue-Green Gotchas
Failure modes that make a 'simple' traffic switch not so simple.
- Sticky sessions- Users mid-session on blue get dropped or re-authenticated if session state isn't shared/externalized (e.g. Redis-backed sessions)
- Cold caches- Green's in-process caches, JIT-warmed code paths, and connection pools start cold, causing a latency spike right after cutover
- Third-party webhooks/callbacks- External services with hardcoded green/blue URLs or IP allowlists can silently break on cutover
- Shared queues- If blue and green both consume from the same queue/topic, ensure message schemas are compatible in both directions
- Double licensing/infra cost- Running two full production-sized environments simultaneously doubles compute cost for the overlap window
- Long-lived connections- WebSockets, gRPC streams, and DB connection pools don't get cut over instantly; they linger until the client reconnects
- Background jobs/cron- Ensure only ONE color's scheduler is active, or duplicate/conflicting job runs occur during the overlap window
Keep the blue environment running (not torn down) for a defined soak period after cutover — instant rollback is only instant if the old environment is still warm and ready to receive traffic.