Deployment Practice — Canary Rollout — CI/CD with GitHub Actions

What You'll Build

You will implement a three-step canary rollout for CricketPulse using GitHub Actions and Kubernetes replica-proportion traffic splitting — without requiring a full service mesh. The workflow deploys a new version to a canary Deployment, waits for health validation at each weight step (5% → 25% → 100%), and promotes or rolls back based on a simulated metric check. You will deliberately inject a failure at the 25% step to observe the automated rollback logic, then fix the issue and observe a successful promotion to 100%. This exercise uses replica-proportional traffic splitting achievable with standard Kubernetes Services, making it runnable in any standard cluster including kind or k3s.

Analogy🏏Cricket

🏏 Think of it like cricket: This exercise is building the complete match management infrastructure: the pitch is prepared (dev), the warm-up match is played (staging approval), and only after the selectors sign off does the Test match begin (production). You'll experience being both the deployer (pushing the commit that initiates the pipeline) and the approver (clicking the approval button in GitHub's reviewer interface) — understanding both sides of the gate is essential for designing gates that are genuinely useful rather than bureaucratic obstacles.

Prerequisites

A Kubernetes cluster accessible from GitHub Actions — k3s, kind, AWS EKS, GKE, or AKS. Alternatively use echo-based mock deploy steps if no cluster is available.
kubectl configured and the kubeconfig stored as `KUBECONFIG_STAGING` in GitHub Secrets as a base64-encoded string.
The container image from Lesson 12 in `ghcr.io` — or a simple mock image for the exercise.
Understanding of Argo Rollouts concepts from Lesson 19 and Kubernetes Deployment basics from Lesson 17.
Node.js installed locally to tag a 'broken' image variant for testing the rollback scenario.

Setup & Project Structure

Deploy the stable version of CricketPulse to the cluster with two separate Deployments sharing one Service. The canary-stable Deployment serves the majority of traffic via more replicas; the canary-new Deployment serves the canary percentage via fewer replicas. This approximation works without a service mesh — traffic distribution is proportional to replica count.

Analogy🏏Cricket

🏏 Think of it like cricket: configuring the three environments and their protection rules in the GitHub UI before writing any workflow is like agreeing the match's playing conditions with the officials before the toss — not mid-innings. Just as environment protection rules live entirely in settings and there is no YAML for them, the tournament's rules on reviews, powerplays, and appeals are fixed by the governing body off the field, not written into a batsman's technique. Just as the workflow simply names an environment and GitHub enforces whatever rules are configured, a player simply takes the field and the umpires enforce the pre-agreed conditions — the player doesn't carry the rulebook, the officials do. The payoff: settle the playing conditions once, in the right place, and every delivery afterwards is automatically governed by them without cluttering the play itself.

bash

# Create Kubernetes manifests for replica-proportion canary
mkdir -p k8s/canary

# Stable deployment (20 replicas initially)
cat > k8s/canary/deployment-stable.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cricketpulse-stable
  namespace: cricketpulse-staging
spec:
  replicas: 20
  selector:
    matchLabels: { app: cricketpulse, track: stable }
  template:
    metadata:
      labels: { app: cricketpulse, track: stable }
    spec:
      containers:
        - name: cricketpulse
          image: ghcr.io/YOUR_USERNAME/cricketpulse:stable
          ports: [{ containerPort: 3000 }]
          readinessProbe:
            httpGet: { path: /health, port: 3000 }
EOF

# Canary deployment (0 replicas initially)
cat > k8s/canary/deployment-canary.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cricketpulse-canary
  namespace: cricketpulse-staging
spec:
  replicas: 0
  selector:
    matchLabels: { app: cricketpulse, track: canary }
  template:
    metadata:
      labels: { app: cricketpulse, track: canary }
    spec:
      containers:
        - name: cricketpulse
          image: ghcr.io/YOUR_USERNAME/cricketpulse:canary
          ports: [{ containerPort: 3000 }]
          readinessProbe:
            httpGet: { path: /health, port: 3000 }
EOF

# Single Service routing to both stable and canary pods
cat > k8s/canary/service.yaml << 'EOF'
apiVersion: v1
kind: Service
metadata:
  name: cricketpulse
  namespace: cricketpulse-staging
spec:
  selector:
    app: cricketpulse   # Routes to ALL pods with app=cricketpulse (both tracks)
  ports:
    - port: 80
      targetPort: 3000
EOF

echo 'Kubernetes canary manifests created'

Step 1 — Foundation

Create the workflow with the build step and the 5% canary deployment. The canary image is built and pushed, then deployed as a single replica (1 of 20 total = ~5%). Health checks confirm the canary pod is ready before proceeding. At this step, 95% of production traffic is unaffected.

Analogy🏏Cricket

🏏 Think of it like cricket: The warm-up session runs on the hotel practice ground with no spectators, no formal protocols — players take the field and start immediately. The dev environment is the hotel practice ground: automated, immediate, no gatekeeping. Confirming the dev pipeline runs automatically before adding gated environments is like confirming the practice session proceeds smoothly before worrying about the match-day protocols.

yaml

# File: .github/workflows/cricketpulse-canary.yml — Step 1

name: CricketPulse Canary Rollout

on:
  push:
    branches: [main]
  workflow_dispatch:
    inputs:
      canary-image-tag:
        description: 'Image tag for canary version'
        required: false
        type: string

jobs:
  build-canary:
    runs-on: ubuntu-latest
    permissions: { contents: read, packages: write }
    outputs:
      canary-tag: ${{ steps.tag.outputs.tag }}
    steps:
      - uses: actions/checkout@v4
      - id: tag
        run: echo "tag=${{ inputs.canary-image-tag || github.sha }}" >> $GITHUB_OUTPUT
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with: { registry: ghcr.io, username: ${{ github.actor }}, password: ${{ secrets.GITHUB_TOKEN }} }
      - uses: docker/build-push-action@v5
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}/cricketpulse:${{ steps.tag.outputs.tag }}
          build-args: APP_VERSION=canary-${{ steps.tag.outputs.tag }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

  canary-5pct:
    runs-on: ubuntu-latest
    needs: build-canary
    environment: staging
    steps:
      - uses: actions/checkout@v4
      - uses: azure/setup-kubectl@v4
        with: { version: 'v1.29.0' }
      - name: Configure kubeconfig
        run: |
          mkdir -p ~/.kube
          echo "${{ secrets.KUBECONFIG_STAGING }}" | base64 -d > ~/.kube/config
      - name: Deploy canary image (1 replica = ~5%)
        run: |
          kubectl set image deployment/cricketpulse-canary \
            cricketpulse=ghcr.io/${{ github.repository }}/cricketpulse:${{ needs.build-canary.outputs.canary-tag }} \
            -n cricketpulse-staging
          kubectl scale deployment/cricketpulse-canary --replicas=1 -n cricketpulse-staging
          kubectl scale deployment/cricketpulse-stable --replicas=19 -n cricketpulse-staging
      - name: Wait for canary rollout
        run: kubectl rollout status deployment/cricketpulse-canary -n cricketpulse-staging --timeout=120s
      - name: Confirm canary pod ready
        run: |
          kubectl wait pod -l 'app=cricketpulse,track=canary' \
            --for=condition=Ready -n cricketpulse-staging --timeout=60s
          echo "Canary pod healthy at 5% weight (1/20 replicas)"

Step 2 — Core Logic

Add the 25% step with a simulated metric check and automatic rollback logic. The analysis script checks whether the canary version tag contains 'broken' — a stand-in for a real Prometheus query. If analysis fails, stable replicas are restored to 20, canary is scaled to 0, and the job exits 1 to prevent promotion.

Analogy🏏Cricket

🏏 Think of it like cricket: The warm-up session is complete (dev deployed). Now the match-day protocols begin: the umpires inspect the pitch, the team captains arrive for the toss, the grounds team prepare the outfield. The staging approval is the toss ceremony — a formal, structured pause before the match begins, requiring the participating parties to confirm readiness before play commences.

yaml

# Step 2: Add 25% step with analysis and rollback (append to jobs)

  canary-25pct:
    runs-on: ubuntu-latest
    needs: [build-canary, canary-5pct]
    environment: staging
    steps:
      - uses: actions/checkout@v4
      - uses: azure/setup-kubectl@v4
        with: { version: 'v1.29.0' }
      - name: Configure kubeconfig
        run: |
          mkdir -p ~/.kube
          echo "${{ secrets.KUBECONFIG_STAGING }}" | base64 -d > ~/.kube/config
      - name: Scale to 25% canary weight (5 canary, 15 stable)
        run: |
          kubectl scale deployment/cricketpulse-canary --replicas=5 -n cricketpulse-staging
          kubectl scale deployment/cricketpulse-stable --replicas=15 -n cricketpulse-staging
          kubectl rollout status deployment/cricketpulse-canary -n cricketpulse-staging --timeout=120s
      - name: Simulated metric analysis at 25%
        id: analysis
        run: |
          CANARY_TAG="${{ needs.build-canary.outputs.canary-tag }}"
          echo "Analysing canary metrics for tag: ${CANARY_TAG}"
          # Simulate failure: any tag containing 'broken' fails analysis
          if echo "${CANARY_TAG}" | grep -q 'broken'; then
            echo "ANALYSIS_RESULT=fail" >> $GITHUB_OUTPUT
            echo "ANALYSIS FAILED: simulated error rate exceeded threshold"
          else
            echo "ANALYSIS_RESULT=pass" >> $GITHUB_OUTPUT
            echo "ANALYSIS PASSED: metrics within thresholds"
          fi
      - name: Rollback if analysis failed
        if: steps.analysis.outputs.ANALYSIS_RESULT == 'fail'
        run: |
          echo "=== CANARY ROLLBACK ==="
          kubectl scale deployment/cricketpulse-canary --replicas=0 -n cricketpulse-staging
          kubectl scale deployment/cricketpulse-stable --replicas=20 -n cricketpulse-staging
          echo "Traffic restored to 100% stable. Canary reverted."
          exit 1

Step 3 — Integration & Enhancement

Add full promotion to 100% and a pipeline summary. The promotion only runs if the 25% analysis passed. It scales canary to 20 replicas, waits for rollout, then scales stable to 0 — completing the traffic handover. The new image then becomes the stable baseline for the next deployment cycle.

Analogy🏏Cricket

🏏 Think of it like cricket: After the warm-up (dev) and the pitch inspection ceremony (staging), the Test match finally begins with all the formal protocols: national anthems, player presentations, the coin toss, a final equipment check. Each is a structured delay ensuring everything is ready before the ball is bowled. The 2-minute production wait timer is the Test match's opening ceremony — a deliberate pause giving everyone a final opportunity to abort if something has gone wrong.

yaml

# Step 3: Full promotion and summary (append to jobs)

  promote-100pct:
    runs-on: ubuntu-latest
    needs: [build-canary, canary-25pct]
    environment: staging
    steps:
      - uses: azure/setup-kubectl@v4
        with: { version: 'v1.29.0' }
      - name: Configure kubeconfig
        run: |
          mkdir -p ~/.kube
          echo "${{ secrets.KUBECONFIG_STAGING }}" | base64 -d > ~/.kube/config
      - name: Promote canary to 100%
        run: |
          kubectl scale deployment/cricketpulse-canary --replicas=20 -n cricketpulse-staging
          kubectl rollout status deployment/cricketpulse-canary -n cricketpulse-staging --timeout=300s
          kubectl scale deployment/cricketpulse-stable --replicas=0 -n cricketpulse-staging
          echo "Canary promoted to 100%. Stable scaled to 0."
      - name: Update stable deployment image for next cycle
        run: |
          kubectl set image deployment/cricketpulse-stable \
            cricketpulse=ghcr.io/${{ github.repository }}/cricketpulse:${{ needs.build-canary.outputs.canary-tag }} \
            -n cricketpulse-staging
          echo "Stable updated to promoted image for next canary cycle"

  rollout-summary:
    runs-on: ubuntu-latest
    needs: [canary-5pct, canary-25pct, promote-100pct]
    if: always()
    steps:
      - run: |
          cat >> $GITHUB_STEP_SUMMARY << 'SUMMARY'
          ## CricketPulse Canary Rollout Summary
          | Stage | Result |
          |-------|--------|
          | 5% Canary | ${{ needs.canary-5pct.result }} |
          | 25% Canary | ${{ needs.canary-25pct.result }} |
          | 100% Promotion | ${{ needs.promote-100pct.result }} |

          **Canary tag:** ${{ needs.build-canary.outputs.canary-tag }}
          **Deployer:** ${{ github.actor }}
          SUMMARY

Step 4 — Testing & Verification

Run three test scenarios: a successful full promotion, an automated rollback at 25% (using a 'broken' tag), and verification of replica counts at each stage.

Analogy🏏Cricket

🏏 Think of it like cricket: running the three scenarios — automatic push, manual dispatch with production, and a rejected staging deploy — is like rehearsing how a match unfolds under different calls before the real fixture. Just as an automatic push flows through dev and staging but stops short of production, an ordinary passage of play proceeds freely up to the boundary rope but can't cross into a scoring appeal without review. Just as manual dispatch with production enabled runs the full three-stage pipeline, a captain formally opting to declare unlocks the complete sequence of events that follows. And just as a deliberately rejected staging deployment halts the pipeline, a third umpire ruling not-out on review sends the batsman back and the passage of play simply stops there. Testing all three teaches you how each gate behaves. The payoff: like rehearsing every umpiring outcome in advance, you learn exactly how approvals let a deploy through, how manual triggers open the full path, and how a rejection cleanly stops the release.

bash

# Test scenarios

# Apply k8s manifests to cluster first
kubectl create namespace cricketpulse-staging --dry-run=client -o yaml | kubectl apply -f -
kubectl apply -f k8s/canary/ -n cricketpulse-staging

# SCENARIO 1: Successful rollout
git add . && git commit -m 'feat: canary score refresh algorithm v2' && git push origin main
# Expected: 5% → 25% → 100% all pass
# Replica counts:
#   After 5%:   canary=1,  stable=19
#   After 25%:  canary=5,  stable=15
#   After 100%: canary=20, stable=0

# SCENARIO 2: Automated rollback at 25%
# Build and push an image with 'broken' in the tag:
# docker build -t ghcr.io/YOUR_USERNAME/cricketpulse:broken-v1 . && docker push ...
# Manually dispatch with: canary-image-tag = broken-v1
# Expected: 5% passes → 25% analysis FAILS → rollback (stable=20, canary=0) → job fails

# Verify after rollback:
kubectl get deployments -n cricketpulse-staging
# cricketpulse-stable  20/20
# cricketpulse-canary   0/0

Warning: The replica-proportion traffic splitting in this exercise is an approximation — with 1 canary and 19 stable replicas you get approximately 5% canary traffic assuming kube-proxy distributes connections uniformly. In practice, connection persistence (keep-alive, sticky sessions) can skew the distribution. For production canary deployments requiring precise percentage control, use a service mesh (Istio), Argo Rollouts with Nginx ingress, or a cloud load balancer with weighted target groups.

Extension Challenge: Replace the simulated metric check in the 25% step with a real Prometheus query using the HTTP API. Query `sum(rate(http_requests_total{status=~"5.."}[2m])) / sum(rate(http_requests_total[2m]))` from a Prometheus instance in your cluster via curl, parse the result with jq, and fail the pipeline if the rate exceeds 0.01 (1%). This makes the canary analysis identical to production-grade systems like Flagger.

Replica-proportion traffic splitting approximates canary percentage routing without a service mesh — suitable for exercises and low-precision production scenarios.
The rollback step must scale canary to 0 and stable back to 20, then exit 1 to fail the job and prevent the promotion step from running.
Each canary step is a separate job with `needs:` dependency — if the 25% job fails, the 100% promotion job is automatically skipped by GitHub Actions.
The promoted canary image should be set as the new stable deployment image so the next deployment cycle starts from the current stable baseline.
Always include a summary job with `if: always()` to report the final state of each canary step regardless of whether the rollout succeeded or was aborted.
`workflow_dispatch` with a canary-image-tag input enables testing specific image tags including the rollback scenario without requiring a code change commit.