Lab — rolling updates, rollbacks and health probe tuning

What You'll Build

In this lab you will operate the IPL scorecard Deployment through its full lifecycle of update, failure, and recovery, developing the operational reflexes that distinguish an engineer who runs Kubernetes from one who only deploys to it. You will perform four scenarios: a successful rolling update with annotated change cause and real-time progress monitoring; a simulated failed rollout where a bad image tag causes CrashLoopBackOff and the rollout halts; a rollback to the last stable revision; and a health probe tuning session that modifies `initialDelaySeconds`, `failureThreshold`, and `periodSeconds` on a slow-starting service to eliminate false-positive liveness probe restarts without increasing the real failure detection window beyond an acceptable SLA. Each scenario is designed around a realistic operational situation where the correct action depends on correctly reading the Kubernetes resource hierarchy.

The probe tuning scenario addresses one of the most common misconfigurations in production Kubernetes deployments: liveness probes that are too aggressive for the application's actual startup time, causing Kubernetes to restart perfectly healthy containers during load spikes or cold starts. The probe values in the Deployment manifest from Lesson 16 are conservative starting points; this lab teaches you to measure actual startup time under load, calculate the minimum `initialDelaySeconds` that avoids false positives, and verify that the resulting failure detection window still satisfies the application's availability SLA — the analytical discipline that separates probe values based on measurement from probe values based on guesswork.

Analogy🏏Cricket

🏏 Think of it like cricket: This lab's staging-then-production TLS pipeline is the ICC's pre-tour practice match protocol. Before the Test series proper, the touring team plays a two-day warm-up match against a local state side — not under ICC playing conditions, not with the official match balls, and not counted in official records. The warm-up match validates that the team's batting and bowling combinations work on local pitch conditions before committing to the conditions for the official five-day Test. The Let's Encrypt staging environment is that warm-up match: it issues real certificates signed by a staging CA (not trusted by browsers, like an unofficial match result), validates the complete ACME challenge flow, and confirms that DNS, network, and IAM configurations are correct — all without consuming the production rate limit. Switching to the production issuer is committing to the official Test: the certificate is now signed by Let's Encrypt's trusted CA (officially recognised), but any misconfiguration wastes an official certificate issuance. The certificate rotation simulation is the team testing their emergency substitution protocol — specifically waiting until the 'player' (certificate) is within the renewal window, confirming the automatic replacement process fires correctly, and verifying the new 'player' is ready to take the field. This reveals why the three-phase structure of the lab matters: each phase validates a distinct property of the TLS chain, and proceeding through them in order reduces the risk that a configuration problem discovered in production was one that staging testing would have caught.

Prerequisites

The IPL scorecard platform from M3 Exercise running and verified — all Pods in Ready state in the `ipl-production` namespace before beginning any scenario in this lab.
Two additional image tags pre-built and loaded into the cluster: one that represents a 'slow start' variant with a 20-second startup delay (for probe tuning) and one non-existent tag `ipl-scorecard:bad-tag` (to simulate a failed pull for the rollback scenario).
A terminal window running `kubectl get pods -n ipl-production -w` to observe Pod state transitions in real time throughout all four scenarios.
Prometheus and Grafana running with the kube-state-metrics exporter to observe `kube_deployment_status_replicas_ready` and `kube_pod_container_status_restarts_total` metrics — optional but recommended for the probe tuning scenario.
A text editor open with the Deployment manifest from M3 Exercise for the probe tuning scenario — you will modify `initialDelaySeconds`, `periodSeconds`, and `failureThreshold` values in three measured iterations.

Setup & Project Structure

Verify the baseline state before running any scenario: all Pods must be Ready, the Deployment's rollout history must show the initial deployment, and the ResourceQuota must show sufficient headroom for the rolling update to create surge Pods. Running a scenario against an already-degraded cluster produces compound failures whose root cause is ambiguous.

Analogy🏏Cricket

🏏 Think of it like cricket: setting up a tournament venue is not one job but a strict sequence of jobs, and smart boards hire a professional event crew instead of doing each task by hand. Just as the event crew handles seating, accreditation desks, and broadcast rigging as one coordinated package, Helm installs the Nginx IngressController and cert-manager with all their CRDs, ServiceAccounts, and RBAC in one release instead of dozens of hand-applied manifests. And order matters: just as the DRS cameras and replay screens must be rigged and tested before the third umpire takes his seat — an official with no working replay feed is useless and the review system collapses — cert-manager's CRDs must exist before the controller starts, or it crashes on startup before its CRD admission webhooks ever come online. The payoff: a correctly sequenced, fully provisioned venue — controller infrastructure that comes up cleanly the first time.

bash

# Verify baseline before beginning scenarios

# All Pods ready
kubectl get pods -n ipl-production -o wide
# Expected: all Pods in Running state, READY column shows 1/1 for all

# Deployment shows healthy status
kubectl get deployment ipl-api -n ipl-production
# Expected: READY=2/2, UP-TO-DATE=2, AVAILABLE=2

# Rollout history shows initial deployment
kubectl rollout history deployment/ipl-api -n ipl-production
# Expected: REVISION 1, CHANGE-CAUSE: "initial deployment from M3 exercise"

# ResourceQuota has headroom for rolling update (maxSurge=1 requires one more Pod slot)
kubectl describe resourcequota ipl-production-quota -n ipl-production | grep "count/pods"
# Expected: count/pods: 4/20 — plenty of headroom for 1 surge Pod

# Create the "slow-start" image variant for probe tuning
cat > Dockerfile.slowstart << 'EOF'
FROM localhost:5001/ipl-scorecard:latest
# Add artificial startup delay to simulate slow JVM/Python import time
ENTRYPOINT ["sh", "-c", "echo 'Simulating slow startup...'; sleep 20; uvicorn main:app --host 0.0.0.0 --port 8000"]
EOF
docker build -f Dockerfile.slowstart -t localhost:5001/ipl-scorecard:slow-start .
kind load docker-image localhost:5001/ipl-scorecard:slow-start --name ipl-cluster

echo "Baseline verified. Ready to begin lab scenarios."

# Open a watch terminal in a second window:
# kubectl get pods -n ipl-production -w

Step 1 — Foundation

Perform a successful rolling update with annotated change cause, monitoring the rollout in real time. The annotate-before-apply discipline is critical: `kubernetes.io/change-cause` must be set before the rollout begins so that `kubectl rollout history` shows a meaningful description rather than an empty string. Watch the Pod state transitions — old Pods moving to Terminating, new Pods progressing through ContainerCreating → Running → Ready — to confirm that `maxUnavailable: 0` keeps two Ready Pods available at all times throughout the transition.

Analogy🏏Cricket

🏏 Think of it like cricket: before the first official fixture, a serious team plays a full-dress practice match — real pitch, real umpires, real match conditions — knowing the result won't appear in any league table. Just as that practice match earns no points but proves the batting order, bowling plans, and fielding drills actually work under match pressure, the Let's Encrypt staging certificate won't be trusted by any browser but proves that the ACME challenge completes, DNS resolves correctly, and the IngressController routes traffic as designed. Just as a coach would never debut an untested game plan in a televised final, you never point at the production issuer first — a failed live attempt costs far more than a failed rehearsal. The payoff: every moving part of the certificate pipeline validated cheaply, so the later switch to production is a formality rather than a gamble.

bash

# Scenario 1: Successful rolling update with real-time monitoring

# ── Step 1: Annotate BEFORE applying ──────────────────────────────────────
kubectl annotate deployment/ipl-api   kubernetes.io/change-cause="feat: Scenario 1 — update to verify rolling mechanics"   -n ipl-production --overwrite

# ── Step 2: Trigger rolling update by changing the image ──────────────────
# (In practice this would be a new digest — we re-apply same image to trigger
# a new rollout while keeping the application functionally identical)
kubectl patch deployment ipl-api   -n ipl-production   -p '{"spec":{"template":{"metadata":{"annotations":{"rollout-trigger":"'$(date +%s)'"}}}}}'

# ── Step 3: Monitor rollout in real time ──────────────────────────────────
kubectl rollout status deployment/ipl-api -n ipl-production --timeout=120s

# ── Step 4: Verify rollout history ────────────────────────────────────────
kubectl rollout history deployment/ipl-api -n ipl-production
# REVISION  CHANGE-CAUSE
# 1         initial deployment from M3 exercise
# 2         feat: Scenario 1 — update to verify rolling mechanics

# ── Step 5: Confirm availability was maintained throughout ─────────────────
# Availability was maintained if maxUnavailable=0 is set
kubectl describe deployment ipl-api -n ipl-production | grep "Max Unavailable"
# Expected: Max Unavailable: 0 — no Pod was ever removed before its replacement was Ready

echo "Scenario 1 complete: rolling update succeeded with zero downtime ✓"

Step 2 — Core Logic

Simulate a failed rollout: update the Deployment to reference a non-existent image tag, watch Pods enter ImagePullBackOff and then ErrImagePull state, observe the rollout halt as the progressDeadlineSeconds timer approaches, then execute a rollback to the previous revision and confirm the Deployment returns to healthy status. The key diagnostic skill is distinguishing ImagePullBackOff (a transient registry authentication or network failure that may self-resolve) from ErrImagePull (a permanent 404 error indicating the tag does not exist) — a distinction that determines whether the correct response is to wait, to fix the registry credentials, or to roll back.

Analogy🏏Cricket

🏏 Think of it like cricket: once the practice match proves the plans work, the team steps into the official fixture — and everything must now count for real. Just as the practice-game scorecard is torn up so the official scorers start a fresh book, the staging certificate Secret is deleted so cert-manager issues a fresh production certificate rather than serving the old untrusted one. Just as the umpires formally verify the match ball and playing conditions before play begins, you verify the production certificate against the system certificate store — curl without -k must succeed. Then comes rehearsing the handover: just as a club renews a key player's contract well before it expires so there is never a day without him under contract, patching renewBefore to 89 days forces cert-manager to renew the 90-day certificate almost immediately, proving rotation happens automatically long before expiry. The payoff: trusted TLS in production plus demonstrated automatic renewal — no midnight expiry emergencies.

bash

# Scenario 2: Failed rollout and recovery via rollback

# ── Step 1: Inject a bad image tag ────────────────────────────────────────
kubectl annotate deployment/ipl-api   kubernetes.io/change-cause="INTENTIONAL FAILURE: bad image tag for lab"   -n ipl-production --overwrite

kubectl set image deployment/ipl-api   ipl-api=localhost:5001/ipl-scorecard:tag-does-not-exist   -n ipl-production

# ── Step 2: Observe the failure progression ────────────────────────────────
kubectl get pods -n ipl-production -w
# New Pod enters: Pending → ContainerCreating → ErrImagePull → ImagePullBackOff
# Old Pods remain Running (maxUnavailable=0: old Pods not removed until new are Ready)

# Diagnose the failure
kubectl describe pod   $(kubectl get pod -l app=ipl-api -n ipl-production     --field-selector=status.phase=Pending -o name | head -1)   -n ipl-production | tail -20
# Events show: Failed to pull image: tag does not exist

# ── Step 3: Check rollout status (shows stuck, not failed yet) ─────────────
kubectl rollout status deployment/ipl-api -n ipl-production
# Waiting for deployment "ipl-api" rollout to finish...
# (will eventually say 'has timed out' after progressDeadlineSeconds=600s)

# Don't wait 10 minutes — roll back immediately once ErrImagePull is confirmed
# ── Step 4: Rollback to last stable revision ──────────────────────────────
kubectl rollout undo deployment/ipl-api -n ipl-production
# deployment.apps/ipl-api rolled back

# ── Step 5: Verify recovery ────────────────────────────────────────────────
kubectl rollout status deployment/ipl-api -n ipl-production --timeout=60s
# deployment "ipl-api" successfully rolled out

kubectl get pods -n ipl-production
# All Pods Running and Ready again ✓

kubectl rollout history deployment/ipl-api -n ipl-production
# Shows the bad revision AND the rollback entry in history

echo "Scenario 2 complete: failed rollout detected and rolled back ✓"

Step 3 — Integration & Enhancement

Tune the liveness probe for the slow-start image variant using a three-iteration measurement approach: deploy with aggressive probe values that trigger false positives, observe the restart count, calculate the correct `initialDelaySeconds` from measured startup time, redeploy with corrected values, and verify that the restart count drops to zero while a genuine liveness failure still triggers a restart within the SLA window. This iterative measurement-and-adjustment workflow is the professional approach to probe configuration — not guessing at values, but measuring the application's behaviour and deriving settings from data.

Analogy🏏Cricket

🏏 Think of it like cricket: on the eve of a tournament the venue runs a full walk-through — a spectator's journey from the car park, through the ticket gate, to the correct stand, with stewards redirecting anyone who wanders toward the wrong entrance. Just as that walk-through exercises every link in the chain, the end-to-end check verifies DNS resolution, the TLS handshake with the production certificate, routing to the correct backend service, and the HTTPS redirect that steers plain-HTTP stragglers to the secure entrance. Just as gate staff read the stand printed on each ticket and route spectators accordingly, host-based routing reads the hostname and sends pgAdmin traffic to its own backend while API traffic goes to the API. And just as an all-venue tournament pass admits its holder to every ground without a separate ticket per stadium, the wildcard certificate covers both subdomains with a single credential. The payoff: one entry point, correctly securing and routing every kind of visitor.

bash

# Scenario 3: Health probe tuning for a slow-starting application

# ── Iteration 1: Deploy with aggressive probes (will cause false positives) ──
kubectl annotate deployment/ipl-api   kubernetes.io/change-cause="lab: probe tuning iteration 1 — aggressive probes"   -n ipl-production --overwrite

kubectl patch deployment ipl-api -n ipl-production -p '{
  "spec": {
    "template": {
      "spec": {
        "containers": [{
          "name": "ipl-api",
          "image": "localhost:5001/ipl-scorecard:slow-start",
          "livenessProbe": {
            "httpGet": {"path": "/health", "port": 8000},
            "initialDelaySeconds": 5,   # too short: 20s startup > 5s delay
            "periodSeconds": 10,
            "failureThreshold": 2       # kills after only 2 failures (20s)
          }
        }]
      }
    }
  }
}'

# Wait 60 seconds, then observe false restarts
sleep 60
kubectl get pod -l app=ipl-api -n ipl-production
# Expected: RESTARTS column shows 2-3 — liveness killed the container
# before startup completed, triggering CrashLoopBackOff

# ── Measure actual startup time ────────────────────────────────────────────
kubectl logs -l app=ipl-api -n ipl-production --previous | grep "Uvicorn running"
# Find timestamp difference: sleep 20 + uvicorn startup ≈ 22-25 seconds

# ── Iteration 2: Apply corrected probe values based on measurement ──────────
kubectl annotate deployment/ipl-api   kubernetes.io/change-cause="lab: probe tuning iteration 2 — measured values"   -n ipl-production --overwrite

kubectl patch deployment ipl-api -n ipl-production -p '{
  "spec": {
    "template": {
      "spec": {
        "containers": [{
          "name": "ipl-api",
          "image": "localhost:5001/ipl-scorecard:slow-start",
          "livenessProbe": {
            "httpGet": {"path": "/health", "port": 8000},
            "initialDelaySeconds": 35,  # measured startup (25s) + 10s margin
            "periodSeconds": 15,        # check every 15s after initial delay
            "failureThreshold": 3       # tolerate 3 failures before restart (45s window)
          }
        }]
      }
    }
  }
}'

# ── Verify: no false restarts, genuine failure still detected ──────────────
sleep 120
RESTARTS=$(kubectl get pod -l app=ipl-api -n ipl-production   -o jsonpath='{.items[0].status.containerStatuses[0].restartCount}')
echo "Restart count after probe tuning: $RESTARTS"
[ "$RESTARTS" -eq "0" ] && echo "No false positives ✓" || echo "Still false positives — increase initialDelaySeconds"

# Detection window: initialDelaySeconds=35 means first check at 35s
# failureThreshold=3 × periodSeconds=15 = 45s to detect a genuine failure
# Total worst-case detection time: 35s + 45s = 80 seconds — acceptable for this SLA

echo "Scenario 3 complete: probe tuning eliminates false positives ✓"

# Reset to standard image for clean lab completion
kubectl set image deployment/ipl-api   ipl-api=localhost:5001/ipl-scorecard:latest   -n ipl-production
kubectl rollout status deployment/ipl-api -n ipl-production

Step 4 — Testing & Verification

Run the final lab checklist confirming all four scenarios completed successfully: the rollout history shows the expected revision sequence, the Deployment is healthy after rollback, the probe configuration no longer causes false restarts, and the application serves correct API responses. The lab is complete when all checklist items pass — and more importantly, when you can explain the operational reasoning behind each scenario without consulting the lab instructions.

Analogy🏏Cricket

🏏 Think of it like cricket: at the midpoint of a season, a good head coach doesn't just check the points table — he traces how every department produced it: the academy that developed the players, the selectors who picked the squad, and the matchday operations that got the team onto the field. Just as that review makes the whole club visible as one system rather than isolated departments, the M1–M4 architecture summary traces each component's role — the container image layer where the application is built and packaged, the Kubernetes workload layer where Deployments, StatefulSets, and autoscaling run it, and the external access layer where Ingress and TLS expose it to the world. Just as ticking the final checklist confirms match-readiness, completing the lab checklist confirms every layer actually works together. The payoff: you can explain the entire platform end to end — the mark of real understanding, and the foundation M5's security work builds on.

bash

# Final lab verification checklist

echo "=== M3 Lab — Rolling Updates and Probe Tuning Verification ==="

# 1. Rollout history shows all scenario revisions
echo -n "1. Rollout history complete:          "
REVISIONS=$(kubectl rollout history deployment/ipl-api -n ipl-production   | grep -v "REVISION" | wc -l)
[ "$REVISIONS" -ge "4" ] && echo "PASS ✓ ($REVISIONS revisions)" || echo "FAIL ✗"

# 2. Deployment is healthy after rollback
echo -n "2. Deployment healthy post-rollback:  "
READY=$(kubectl get deployment ipl-api -n ipl-production   -o jsonpath='{.status.readyReplicas}')
DESIRED=$(kubectl get deployment ipl-api -n ipl-production   -o jsonpath='{.spec.replicas}')
[ "$READY" -eq "$DESIRED" ] && echo "PASS ✓ ($READY/$DESIRED ready)" || echo "FAIL ✗"

# 3. No containers in CrashLoopBackOff (probe tuning resolved false restarts)
echo -n "3. No CrashLoopBackOff pods:          "
CRASH=$(kubectl get pods -n ipl-production   --field-selector=status.phase=Running   -o jsonpath='{.items[*].status.containerStatuses[*].state.waiting.reason}'   | grep -c "CrashLoopBackOff" || true)
[ "$CRASH" -eq "0" ] && echo "PASS ✓" || echo "FAIL ✗ ($CRASH crashing)"

# 4. API responds correctly after all scenario transitions
echo -n "4. API serves correct responses:      "
kubectl port-forward service/ipl-api 8080:8000 -n ipl-production &
PF_PID=$!
sleep 3
STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health)
kill $PF_PID 2>/dev/null
[ "$STATUS" -eq "200" ] && echo "PASS ✓ (HTTP 200)" || echo "FAIL ✗ (HTTP $STATUS)"

# 5. PDB protects the Deployment (verify it exists)
echo -n "5. PodDisruptionBudget present:       "
kubectl get pdb -n ipl-production 2>/dev/null | grep -q "ipl-api"   && echo "PASS ✓" || echo "FAIL ✗ — apply PDB from Lesson 16"

echo "=== Lab complete ==="
echo ""
echo "Operational skills demonstrated:"
echo "  - Rolling update with annotated change cause ✓"
echo "  - Failed rollout detection (ErrImagePull) ✓"
echo "  - Rollback to stable revision ✓"
echo "  - Probe tuning from measurement, not guesswork ✓"

Warning: Never set `livenessProbe.failureThreshold` to 1 in production. A single probe failure is not evidence of a genuine application liveness failure — it may be a momentary network hiccup, a brief garbage collection pause, or a probe executor scheduling delay on a loaded node. A `failureThreshold` of 1 will restart containers during normal transient network conditions, creating unnecessary downtime and masking real performance issues behind a noisy restart count. The minimum recommended `failureThreshold` for production is 3, giving the probe three consecutive failures over `failureThreshold × periodSeconds` before restarting, which is enough to filter transient blips while still detecting genuine application hangs within an acceptable window.

Extension Challenge: Implement a canary deployment for the IPL API using Kubernetes label selectors and weighted traffic splitting. Create a second Deployment named `ipl-api-canary` with one replica and the new image tag, using the same labels as the primary Deployment so both Deployments' Pods receive traffic from the existing Service. The canary receives approximately 25% of traffic (1 of 4 total Pods) without any changes to the Service definition. Monitor the canary's error rate separately using a label selector on the Pod label `version: canary` in Prometheus, and only proceed to a full rollout if the canary's error rate matches the baseline. This is the manual equivalent of the Argo Rollouts operator's canary strategy, which M4 will cover with automated traffic weight management.

Always annotate `kubernetes.io/change-cause` before every rollout — the annotation becomes the rollout history entry that enables post-mortem root cause analysis by correlating incident timelines with specific deployment revisions.
ErrImagePull indicates a permanent 404 (tag not found) while ImagePullBackOff indicates exponential backoff after any pull failure — ErrImagePull warrants immediate rollback, while ImagePullBackOff may resolve with a registry fix.
Kubernetes never automatically rolls back a failed rollout — it halts progress after `progressDeadlineSeconds` and marks the Deployment Failed, requiring an explicit `kubectl rollout undo` decision from an operator.
Probe tuning requires measurement: deploy with a watch on `kube_pod_container_status_restarts_total`, observe actual startup time, set `initialDelaySeconds` to measured startup + margin, and verify zero restarts before considering the configuration correct.
The detection window for a genuine liveness failure is `initialDelaySeconds + (failureThreshold × periodSeconds)` — balance this against the desired time-to-restart SLA to avoid both false positives and delayed genuine failure detection.
A `failureThreshold` of 1 causes restarts on any transient probe failure; production minimum is 3, providing three consecutive check failures over the period window as evidence before triggering a container restart.