Resilience Practice — Run a Game Day — DevSecOps & Site Reliability Engineering

What You'll Build

In this exercise you will plan and execute a structured Game Day for CricketPulse with two fault injection experiments: pod failure on the score API, and latency injection on the payment service. You will deploy Chaos Mesh, define experiment CRDs, run experiments with Grafana monitoring active, document findings, and produce a Game Day report with action items. The exercise is designed to run in a staging Kubernetes cluster.

Analogy🏏Cricket

🏏 Think of it like cricket: This is a full-match simulation in the nets — a structured practice match with umpires, scorers, and a target, played under match conditions so the team builds muscle memory for the real thing. The coach throws curveballs to test the team's response to unexpected events. After the simulation, the team watches the footage and identifies improvements for the next real match.

Prerequisites

Lessons 17, 18, and 19 completed.
A Kubernetes staging cluster with CricketPulse deployed (score-api and payment-service).
Horizontal Pod Autoscaler configured on score-api (min: 2, max: 10 replicas).
Grafana dashboard from the Observability course showing error rate, p99 latency, and pod count.
Chaos Mesh installed (helm install chaos-mesh chaos-mesh/chaos-mesh).
A circuit breaker configured on the payment service (e.g., Resilience4j or Istio retry policy).

Step 1 — Pre-Game Day Setup

Verify all prerequisite systems are in place before starting the Game Day clock. A pre-flight checklist prevents discovering missing infrastructure mid-experiment.

Analogy🏏Cricket

🏏 Think of it like cricket: The morning-of-match inspection — pitch, boundary rope, stumps, DRS calibration, scoreboard. Everything must be confirmed before the toss. A Game Day pre-flight is that inspection: confirm all systems are ready before the first fault is injected.

bash

#!/bin/bash
# Pre-Game Day preflight checklist
set -e

echo "=== CricketPulse Game Day Preflight ==="

# 1. Confirm Chaos Mesh is running
echo "Checking Chaos Mesh..."
kubectl get pods -n chaos-testing -l app.kubernetes.io/instance=chaos-mesh   --field-selector=status.phase=Running | grep -c Running
# Expected: >= 3 pods running

# 2. Confirm CricketPulse services are healthy
echo "Checking CricketPulse services..."
kubectl get pods -n cricketpulse --field-selector=status.phase=Running
for deploy in score-api payment-service checkout-api; do
  kubectl rollout status deployment/$deploy -n cricketpulse --timeout=30s
  echo "  $deploy: OK"
done

# 3. Confirm steady state metrics (all should be in green range)
echo "Checking steady state..."
curl -s "http://prometheus:9090/api/v1/query"   --data-urlencode 'query=rate(http_requests_total{status=~"5..",namespace="cricketpulse"}[5m])'   | python3 -c "import json,sys; d=json.load(sys.stdin); print('Error rate:', d['data']['result'])"
# Expected: empty result or 0.0 (no errors)

# 4. Confirm HPA is configured on score-api
kubectl get hpa score-api -n cricketpulse
# Expected: MINPODS=2, MAXPODS=10, CURRENT=2

# 5. Confirm Grafana dashboard is accessible
curl -s -o /dev/null -w "%{http_code}" http://grafana:3000/api/health
# Expected: 200

echo "=== Preflight PASSED — Ready for Game Day ==="

Step 2 — Experiment 1: Score API Pod Failure

Hypothesis: if one score-api pod is deleted, the HPA creates a replacement within 60 seconds and error rate remains below 1%. Deploy the experiment CRD, observe the response, and record actual outcomes.

Analogy🏏Cricket

🏏 Think of it like cricket: Scenario 1 — opening batter is retired hurt in over 3. Hypothesis: the next batter comes in within 3 minutes and the run rate holds above 6. You inject the fault (remove the batter), observe what actually happens (how long does it take, does the run rate hold), and record the result.

bash

# Define Experiment 1: Pod Failure
cat > experiment-01-pod-failure.yaml << 'EOF'
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: score-api-pod-failure
  namespace: cricketpulse
spec:
  action: pod-kill
  mode: one                    # kill exactly one pod
  selector:
    namespaces: [cricketpulse]
    labelSelectors:
      app: score-api
  duration: "30s"              # experiment duration (pod stays killed until HPA recreates)
  gracePeriod: 0               # kill immediately, no graceful shutdown
EOF

# T+00:00 — Record steady state
echo "BASELINE at $(date -u)"
kubectl get pods -n cricketpulse -l app=score-api
# Record replica count: should be 2

# Take Grafana screenshot: error rate and p99 latency panels

# T+05:00 — Apply experiment
kubectl apply -f experiment-01-pod-failure.yaml
echo "EXPERIMENT STARTED at $(date -u)"

# T+05:00 - T+07:00 — OBSERVE (do not fix, do not panic, just watch)
# Watch pod count in real time
watch kubectl get pods -n cricketpulse -l app=score-api

# Watch Grafana: does error rate spike? If yes: how high, for how long?
# Watch Grafana: does HPA show replica count changing?
# Watch Grafana: does p99 latency increase?

# Expected observations:
# T+05:00: one pod shows Terminating
# T+05:05: pod count drops to 1
# T+05:15: HPA detects low replica count, creates new pod
# T+05:45: new pod shows Running and passes readiness probe
# T+06:00: error rate returns to <0.1%
# Error rate during gap: should be <1% if connection draining works

# T+07:00 — Record post-experiment state
echo "POST-EXPERIMENT at $(date -u)"
kubectl get pods -n cricketpulse -l app=score-api
# Expected: 2 Running pods again

python

# Observations log for Experiment 1 (fill in during the experiment)
EXP1_LOG = """
## Experiment 1: Score API Pod Failure

**Hypothesis**: HPA replaces pod within 60s; error rate < 1% during gap

**Baseline** (T+00:00):
- Replica count: [fill]
- Error rate: [fill]%
- P99 latency: [fill]ms

**Fault Injected** (T+05:00):
- Action: kubectl apply experiment-01-pod-failure.yaml
- Confirmed pod killed: [fill pod name]

**Observations**:
- T+05:05: pod count dropped to [fill]
- T+05:[fill]: HPA triggered — new pod creation started
- T+05:[fill]: new pod Running and ready
- Error rate during gap: [fill]% (peak)
- P99 latency during gap: [fill]ms (peak)
- Duration of degradation: [fill] seconds

**Hypothesis result**: PASSED / DEVIATED
  If deviated: [describe what was different]

**Findings**:
- [fill: what did you learn about your system?]
"""

Step 3 — Experiment 2: Payment Service Latency Injection

Hypothesis: injecting 1-second latency on the payment service triggers the circuit breaker within 3 retries, and the checkout API returns a graceful degraded response (503 with a retry-after header) rather than queuing indefinitely.

Analogy🏏Cricket

🏏 Think of it like cricket: Scenario 2 — the opposition brings on a slow-medium bowler (latency injection) instead of the expected fast bowler. Hypothesis: the batter adjusts their footwork (circuit breaker activates) within 3 deliveries and plays the correct shot (graceful degradation). If the batter freezes and gets out (connection pool exhaustion), the hypothesis fails.

bash

# Define Experiment 2: Payment Latency Injection
cat > experiment-02-payment-latency.yaml << 'EOF'
apiVersion: chaos-mesh.org/v1alpha1
kind: NetworkChaos
metadata:
  name: payment-latency
  namespace: cricketpulse
spec:
  action: delay
  mode: all                    # affect all payment-service pods
  selector:
    namespaces: [cricketpulse]
    labelSelectors:
      app: payment-service
  delay:
    latency: "1000ms"          # 1 second added latency
    jitter: "100ms"
    correlation: "100"         # consistent latency (not random)
  direction: from              # delay incoming requests to payment service
  duration: "3m"
EOF

# Wait 5 minutes between experiments for system to fully recover
echo "Waiting 5 minutes for recovery from Experiment 1..."
sleep 300

# T+15:00 — Record steady state for Experiment 2
echo "BASELINE (Exp 2) at $(date -u)"
# Record checkout error rate, checkout p99 latency

# T+20:00 — Apply experiment
kubectl apply -f experiment-02-payment-latency.yaml
echo "EXPERIMENT 2 STARTED at $(date -u)"

# T+20:00 - T+23:00 — OBSERVE
# Watch Grafana: checkout p99 latency — does it climb by ~1 second?
# Watch Grafana: does checkout error rate increase?
# Watch Grafana: circuit breaker state (if exposed as metric)
# Watch logs: kubectl logs -l app=checkout-api -n cricketpulse -f | grep -i "circuit\|timeout\|retry"

# T+23:00 — Remove experiment
kubectl delete -f experiment-02-payment-latency.yaml
echo "EXPERIMENT 2 ENDED at $(date -u)"

# Verify system recovery (error rate drops within 30s of experiment removal)

Step 4 — Produce the Game Day Report

Compile all observations into a Game Day report document. This is the primary output of the exercise — a structured record of what was tested, what happened, and what will be fixed.

Analogy🏏Cricket

🏏 Think of it like cricket: A training match teaches the squad nothing lasting until someone writes the match report. Just as the team analyst gathers every observation from the session — how long the replacement batter took to settle, whether the run rate held, how the field responded to each scenario — and compiles them into one structured document, you consolidate all your Game Day observations into a single report. Just as that report is the real deliverable of the practice match, not the drills themselves — it is what the coaching staff read, act on, and carry into selection — the Game Day report is the primary output of the exercise, the record of what was tested, what actually happened, and what will be fixed. Just as a report with no clear next steps leaves the same weaknesses unaddressed for the next match, a report without action items produces no improvement. The payoff: writing it down converts a few hours of controlled failure into durable organisational learning the whole team can use.

python

# Game Day report template
GAMEDAY_REPORT = """
# CricketPulse Game Day Report — {date}

## Participants
- IC: [your name]
- Technical Lead: [name]
- Observer: [names]

## Steady State Definition
- Score API success rate: >99%
- Score API p99 latency: <500ms
- Checkout success rate: >99.5%
- Checkout p99 latency: <1000ms

## Experiment 1: Score API Pod Failure
**Hypothesis**: HPA replaces pod within 60s; error rate <1%
**Result**: PASSED / DEVIATED
**Actual replacement time**: [fill] seconds
**Peak error rate during gap**: [fill]%
**Findings**: [fill]

## Experiment 2: Payment Service Latency (1s injection)
**Hypothesis**: Circuit breaker trips within 3 retries; checkout returns 503 gracefully
**Result**: PASSED / DEVIATED
**Circuit breaker activated**: Yes/No at T+[fill] seconds
**Peak checkout error rate**: [fill]%
**Checkout p99 latency during experiment**: [fill]ms
**Findings**: [fill]

## Action Items
| # | Finding | Action | Owner | Due |
|---|---------|--------|-------|-----|
| A1 | [finding from Exp 1] | [specific fix] | [name] | [date] |
| A2 | [finding from Exp 2] | [specific fix] | [name] | [date] |

## What Worked Well
- [fill: at least 2 items]

## What to Improve
- [fill: at least 2 items]

## Next Game Day
- Date: [4-8 weeks from today]
- Proposed scenarios: [fill]
"""

# Save to: gamedayreports/2025-06-03-cricketpulse.md

Verify Your Work

Review your Game Day against this checklist. A successful Game Day meets all six criteria.

Analogy🏏Cricket

🏏 Think of it like cricket: A practice session is worthless until you grade it against clear benchmarks. Just as a batting coach scores a net session on measurable targets — did the batter rotate strike, was the trigger movement quick enough, did they leave the balls outside off — you review the drill against five concrete criteria rather than a vague sense that 'it went okay'. Just as the coach times how fast the batter read the length (time to detect, target under ten minutes) and how quickly they adjusted their shot (time to contain, target under twenty), you measure detection and containment against fixed thresholds. Just as a session only passes if every benchmark is met, not most of them, a passing drill must satisfy all five: fast detection, fast containment, status page updated, a systemic root cause, and at least three owned action items. The payoff: honest scoring against a rubric shows exactly which reflex to sharpen before the real match, rather than leaving you falsely confident.

bash

# Game Day success criteria

# 1. Preflight passed — all services healthy before experiments started
# Yes / No

# 2. Experiments were run in sequence with 5-minute recovery gap between them
# Yes / No

# 3. Baseline metrics recorded before each experiment (screenshots or values)
# Yes / No

# 4. All experiments had an automatic duration (no manual "I'll stop it later")
# Yes / No: experiment-01 duration: 30s, experiment-02 duration: 3m

# 5. Game Day report produced with:
#   - Hypothesis for each experiment
#   - Actual observations vs hypothesis
#   - At least one action item per experiment that deviated
# Yes / No

# 6. System returned to steady state after both experiments
# Verify: kubectl get pods -n cricketpulse (all Running)
# Verify: error rate back to <0.1% in Grafana

# Scoring:
# 6/6: Game Day complete — schedule next one in 4-8 weeks
# 4-5/6: Good — note gaps and address before next Game Day
# <4: Review experiment structure from Lesson 18 and repeat

If either experiment causes a cascade that does not self-recover within 5 minutes of the experiment ending, do not run Experiment 2 in the same session. Restore the system to steady state first, document the cascade as a finding, and investigate the root cause before scheduling the next Game Day. A cascading failure during a Game Day is a learning opportunity, not a failure of the Game Day process.

After your first Game Day, review the experiments against your runbooks from Module 4. If a Game Day experiment reveals a failure mode that is not covered by any runbook, create a new runbook for it before the next Game Day. Game Day findings and runbook coverage are complementary: Game Days discover failure modes, runbooks enable efficient response when those modes occur in production.

Preflight checklist must pass before starting any Game Day experiment.
Always record baseline metrics before injecting a fault — comparison requires a before state.
Wait for full system recovery between experiments — back-to-back experiments can mask independent findings.
Set duration on all Chaos Mesh experiments — no open-ended experiments that outlast the session.
The Game Day report is the primary output — without documentation, the session produces no organisational learning.