ArgoCD canary rollout with auto-rollback on error-rate alert — CI/CD, GitOps, DevSecOps & Observability

What You'll Build

In this lesson you will deploy the india-squad-api using Argo Rollouts with a canary strategy that automatically tests new versions against a Prometheus error-rate metric before promoting them to full traffic. The rollout uses two analysis steps—one at 10% traffic and one at 50%—that query the canary pods' error rate from Prometheus every minute. If the error rate exceeds 5% twice in a row, the rollout is automatically aborted and all traffic returns to the stable version.

You will also test the auto-rollback mechanism by deploying a version that returns HTTP 500 errors, watching the AnalysisRun detect the elevated error rate, and confirming that the Rollout transitions to Degraded while the stable pods continue serving all production traffic uninterrupted. This end-to-end test is the capstone's verification that the observability layer and the delivery layer are connected.

Analogy🏏Cricket

Think of it like cricket: Picture setting up a remote ground management system for a cricket ground in a new city. First, the infrastructure must be installed: the pitch sensors, the scoreboard network, and the broadcast uplink—equivalent to installing ArgoCD on the EKS cluster. Then, the venue configuration must be committed to the central venue management database: the pitch dimensions, the lighting schedule, the boundary positions—equivalent to committing the Kubernetes manifests to the GitOps repository. Then, the venue must be registered with the central management system, which then automatically enforces the declared configuration at the ground—equivalent to creating the ArgoCD Application that connects the repository to the cluster. When a ground manager moves a boundary rope by hand, the sensors detect the drift and alert the management system to restore the declared position—equivalent to ArgoCD detecting and reverting the manual replica scale. This reveals why the lab sequence matters: you cannot verify GitOps until all three components—operator, repository, and Application—are connected and working together.

Prerequisites

Argo Rollouts installed on the EKS cluster: helm install argo-rollouts argo/argo-rollouts --namespace argo-rollouts --create-namespace.
The kube-prometheus-stack from M5 Lab 35 deployed with the india-squad-api ServiceMonitor scraping metrics from the squad-prod namespace.
The ALB ingress controller installed on EKS with the india-squad-api ingress resource configured, since ALB traffic splitting is required for precise canary weight control.
The Kyverno ClusterPolicy from M3 Lab 21 deployed in Enforce mode, so the Rollout will only admit the Cosign-signed image from Lesson 37's pipeline.
The kubectl Argo Rollouts plugin installed locally for the watch and set-image commands.

Step 1 — Deploy AnalysisTemplate and Rollout

Apply the AnalysisTemplate that queries Prometheus for the canary pods' error rate, and the Rollout resource that defines the six-step canary promotion. The AnalysisTemplate uses the `pod_template_hash` label—injected automatically by Kubernetes on every pod—to scope the Prometheus query to only the canary pods rather than the full deployment, ensuring the stable pods' traffic does not dilute the canary's error rate signal.

Analogy🏏Cricket

🏏 Think of it like cricket: Deploying the AnalysisTemplate and Rollout is like setting up a trial that evaluates only the debutant's own performance, cleanly separated from the seasoned players around them. The Rollout is the graduated debut plan — a fixed six-step programme of increasing exposure — and the AnalysisTemplate is the scoring rubric that queries the live stats to judge the newcomer. The crucial detail is the `pod_template_hash` label, which Kubernetes stamps automatically on every pod for its own bookkeeping, like the unique registration number every player already carries. Just as a selection panel filters the scorecard by that one debutant's registration number so the veterans' runs do not inflate the trial verdict, the AnalysisTemplate filters its Prometheus query by the canary's pod_template_hash so the stable pods' clean traffic cannot dilute the canary's error-rate signal. The payoff: scoping the measurement to only the new version's pods means the trial judges the candidate on its own merits rather than a blurred average across old and new.

yaml

# Argo Rollouts: canary strategy with Prometheus AnalysisTemplate.
# Deploys the signed image from Lesson 37 with automated error-rate gate.

# ── 1. AnalysisTemplate: queries Prometheus for canary error rate ─────────
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: india-squad-error-rate
  namespace: squad-prod
spec:
  args:
    - name: canary-hash
  metrics:
    - name: error-rate
      interval: 1m
      successCondition: result[0] <= 0.01    # pass if error rate <= 1%
      failureCondition: result[0] > 0.05     # fail immediately if > 5%
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus-operated.monitoring.svc.cluster.local:9090
          query: |
            sum(rate(http_requests_total{
              namespace='squad-prod',
              pod_template_hash='{{args.canary-hash}}',
              status=~'5..'
            }[2m]))
            /
            sum(rate(http_requests_total{
              namespace='squad-prod',
              pod_template_hash='{{args.canary-hash}}'
            }[2m]))
---
# ── 2. Rollout: canary strategy referencing the AnalysisTemplate ──────────
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: india-squad-api
  namespace: squad-prod
spec:
  replicas: 5
  selector:
    matchLabels: { app: india-squad-api }
  template:
    metadata:
      labels: { app: india-squad-api }
    spec:
      containers:
        - name: api
          image: 123456789.dkr.ecr.ap-south-1.amazonaws.com/india-squad-api:latest
          ports: [{containerPort: 8000}]
  strategy:
    canary:
      canaryService:  india-squad-api-canary
      stableService:  india-squad-api-stable
      trafficRouting:
        alb:
          ingress: india-squad-ingress
          servicePort: 80
      steps:
        - setWeight: 10
        - pause: {duration: 2m}
        - analysis:
            templates:
              - templateName: india-squad-error-rate
            args:
              - name: canary-hash
                valueFrom:
                  podTemplateHashValue: Latest
        - setWeight: 50
        - pause: {duration: 3m}
        - analysis:
            templates:
              - templateName: india-squad-error-rate
            args:
              - name: canary-hash
                valueFrom:
                  podTemplateHashValue: Latest
        - setWeight: 100

Step 2 — Trigger a Rollout and Monitor

Trigger a new rollout by updating the image reference to the signed image digest produced by Lesson 37's CI pipeline. Use the kubectl Argo Rollouts plugin to watch the rollout progress through its steps in real time. Observe the AnalysisRun being created when the analysis step is reached and the measurements being collected every minute. A successful rollout should take approximately 12 minutes to complete all six steps.

Analogy🏏Cricket

Think of it like cricket: Watching the rollout progress with `kubectl argo rollouts get rollout --watch` is like watching the match scoreboard update after each over. The step sequence—10% weight, 2-minute pause, analysis, 50% weight, 3-minute pause, analysis, 100%—is the match programme: each phase has a defined duration and a performance gate. The AnalysisRun creating measurements every minute is like the scoring system updating the run rate after each delivery. Just as the match commentary shows both the current score and whether the required run rate is achievable, the Argo Rollouts dashboard shows both the current traffic weight and whether the AnalysisRun is trending toward success or failure.

bash

# Verify the canary rollout and test auto-rollback.

# 1. Apply the Rollout and AnalysisTemplate to the cluster
kubectl apply -f analysis-template.yaml
kubectl apply -f rollout.yaml

# 2. Install Argo Rollouts kubectl plugin
# brew install argoproj/tap/kubectl-argo-rollouts
# OR: curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64

# 3. Watch the rollout status in real time
kubectl argo rollouts get rollout india-squad-api -n squad-prod --watch
# Expected: shows current weight, steps progress, and AnalysisRun status

# 4. Trigger a new rollout by updating the image
kubectl argo rollouts set image india-squad-api \
  api=123456789.dkr.ecr.ap-south-1.amazonaws.com/india-squad-api:$(git rev-parse HEAD) \
  -n squad-prod

# 5. Monitor the AnalysisRun during the analysis step
kubectl get analysisrun -n squad-prod --watch
# Expected: AnalysisRun created, measurements taken every minute

# 6. Test auto-rollback by deploying a version that returns HTTP 500
# (In the application code, temporarily add: raise HTTPException(500, 'test'))
# Rebuild and push: the pipeline will still sign it (500s are app errors, not CVEs)
# kubectl argo rollouts set image ... (new SHA with the 500 error)
# Watch: the AnalysisRun should detect error rate > 5% and abort the rollout
# kubectl argo rollouts get rollout india-squad-api -n squad-prod
# Expected: Phase: Degraded, weight returns to 0%, stable pods continue serving

# 7. Verify the rollback
kubectl get rollout india-squad-api -n squad-prod \
  -o jsonpath='{.status.currentStepIndex}'
# Expected: 0 (rolled back to the beginning of the step sequence)

Step 3 — Test Auto-Rollback

Deploy a deliberately broken version of the application that returns HTTP 500 errors for all requests. When the canary receives its 10% traffic allocation and the AnalysisRun begins measuring its error rate, the Prometheus query will return a value above the 5% failure threshold. After two consecutive failure measurements—matching the `failureLimit: 2` configuration—the AnalysisRun transitions to Failed and the Rollout automatically aborts, setting the traffic weight back to 0% and scaling down the canary pods. The stable version continues serving 100% of traffic throughout.

Analogy🏏Cricket

Think of it like cricket: The auto-rollback test is the mandatory equipment failure drill that every ICC-certified ground must pass before hosting an international match. The ground manager deliberately disables the floodlights at 20% power to confirm the monitoring system detects the violation and the automated response correctly escalates to the match referee and initiates the backup power protocol—without the match having to be abandoned. Just as the drill proves the safety systems work before they are needed in a real emergency, the auto-rollback test proves the Argo Rollouts analysis and rollback mechanisms work before a real bad deployment reaches production at full traffic. A platform that has never been tested for rollback behaviour under real failure conditions cannot be trusted to respond correctly when a bad deployment occurs at 3 AM.

Warning: The AnalysisTemplate's Prometheus query uses `pod_template_hash` as a label to scope to canary pods only. This label is automatically set by Kubernetes and its value changes with every new pod template—which is exactly what happens when the Rollout deploys a new image. If you test the query manually in the Prometheus expression browser before a rollout is active, you may get 'no data' because no pods with the active canary hash are running yet. This is correct behaviour: the analysis template is valid; it just has no data to evaluate until the canary pods are running and receiving traffic.

Capstone Tip: Add a `SyncFail` ArgoCD hook that calls the Argo Rollouts abort API when an ArgoCD sync fails before the Rollout reaches a stable state. This ensures that a failed Rollout during a GitOps sync is automatically cleaned up rather than leaving the cluster in an intermediate state where the Rollout is neither progressing nor aborted. The hook is a Kubernetes Job with an ArgoCD annotation: `argocd.argoproj.io/hook: SyncFail`.