What You'll Build
In this exercise you will run a structured 60-minute incident drill for the CricketPulse platform. You will play the role of Incident Commander for a simulated P2 incident — a 25% error rate on the match score API caused by a misconfigured rate limit. You will work through the complete lifecycle: detect, triage, contain, resolve, and produce a postmortem document. The drill is designed to be run solo (simulated) or with a team (live role-play).
Analogy🏏Cricket
🏏 Think of it like cricket: This is a full-match simulation in the nets — a structured practice match with umpires, scorers, and a target, played under match conditions so the team builds muscle memory for the real thing. The coach throws curveballs to test the team's response to unexpected events. After the simulation, the team watches the footage and identifies improvements for the next real match.
Prerequisites
- Lessons 13, 14, and 15 completed.
- A copy of the incident IC checklist from Lesson 13.
- The postmortem template from Lesson 15.
- The CricketPulse Grafana dashboard from the Observability course (or a simulated version).
- A Slack workspace or equivalent for the incident channel simulation.
- 60 uninterrupted minutes for the drill.
Step 1 — Drill Briefing and Setup
Before starting the timer, read the scenario and set up your incident channel. The drill will inject a simulated fault into the system description — you should respond as if it is a real P1/P2 incident, following the IC checklist exactly.
Analogy🏏Cricket
🏏 Think of it like cricket: The batting coach briefs the team before the practice match: 'We are simulating a 180-run chase in 20 overs. The opposition opens with two fast bowlers and a spinner from over 7. Treat this exactly like a real match — no holding back.' The briefing sets the simulation parameters before the timer starts.
python
# DRILL SCENARIO: CricketPulse Match Score API Degradation
# Severity: P2 (you will determine this during triage)
# Injected fault: misconfigured rate limit on /api/match/{id}/score
# Symptoms:
# - 25% of requests to /api/match/{id}/score return HTTP 429
# - Error started approximately 18 minutes ago
# - First reported by a user on Twitter: "CricketPulse score not loading"
# - PagerDuty alert: SlowP99Latency (warning) fired 5 minutes ago
# - HighErrorRate (critical) has NOT fired yet
# Simulated Grafana state (read these as if you are looking at dashboards):
GRAFANA_STATE = {
"error_rate": "25% (429 Too Many Requests)",
"p99_latency": "2.3 seconds (degraded from baseline 0.4s)",
"requests_per_second": "normal (800 RPS)",
"affected_endpoints": ["/api/match/{id}/score"],
"unaffected_endpoints": ["/api/match/{id}/commentary", "/api/checkout"],
"last_deployment": "4 hours ago (minor config update to nginx-configmap)",
"recent_changes": ["nginx-configmap updated: rate_limit_requests_per_second changed"],
"pod_status": "all pods Running, no restarts",
"database_status": "healthy, normal query times",
}
# Start the timer. The drill clock begins NOW.
# You have 60 minutes to complete detection through postmortem.Step 2 — Detect and Triage (Minutes 0-10)
Execute the detection and triage phase of the IC checklist. Determine severity, open the incident channel, and assess the blast radius. Document your decisions as you make them — time-stamping each action.
Analogy🏏Cricket
🏏 Think of it like cricket: The first ten minutes of a run chase are all about reading the situation before swinging. Just as the batter arriving at the crease first assesses how bad it is — the required run rate, wickets in hand, which bowler is on — you determine severity before doing anything else, checking error rate and revenue impact to decide P1 versus P2. Just as the captain calls the field together and opens clear communication so everyone shares one plan, you open the incident channel and post the initial status so responders share one picture. Just as a batter gauges the blast radius of a collapse — is it only the top order, or has the middle order gone too — you assess how far the fault spreads: score API only, or checkout as well. And just as the scorer time-stamps every ball for the official record, you document each decision with a timestamp as you make it, so the timeline writes itself. The payoff: practising a disciplined triage builds the reflex to size up a real incident calmly instead of flailing.
python
# EXPECTED IC ACTIONS (minutes 0-10)
# Minute 0-2: Confirm the incident
# - PagerDuty alert received (SlowP99Latency)
# - Twitter report found via social media monitoring
# - Confirm: curl -s "http://localhost:8000/api/match/IPL-001/score" → 429
# Minute 2-4: Determine severity
# Error rate: 25% → affects significant user subset
# Revenue impact: score API is read-only, no checkout impact → P2
# SLO: error budget at risk but not immediately exhausted → P2
# Minute 4-5: Open incident channel
echo "Creating incident channel: #incident-2025-06-01-score-api-429"
# Post initial message to channel:
INITIAL_MESSAGE = """
[INCIDENT OPENED - 22:04 UTC]
IC: [your name]
Service: CricketPulse Score API
Severity: P2 (confirming)
Symptoms: 25% of /api/match/*/score requests returning 429
Detection: Twitter report + PagerDuty SlowP99Latency alert
Blast radius: Score API only, checkout unaffected, ~800 users/min affected
Next update: 22:19 UTC
"""
# Minute 5-10: Assign roles (solo: you play all roles; team: assign)
# Technical Lead: investigate root cause
# Comms Lead: update status page (P2 → post update)
# IC: coordinate and time-box
# Status page update (P2 triggers update within 15 minutes)
STATUS_UPDATE = """
Investigating: Some users may experience errors loading match scores.
We are actively investigating. Updates every 15 minutes.
"""Step 3 — Contain and Resolve (Minutes 10-35)
Work through the investigation and containment. The simulated system state provides all the information you need — use the diagnostic commands to narrow the cause, then implement the containment fix.
Analogy🏏Cricket
🏏 Think of it like cricket: The tactical analysis phase: study the bowling data, identify the pattern, make the adjustment. Rohit Sharma's batting review doesn't end with 'the ball hit his stumps' — it traces the shot back to the delivery type, the field setting, and the match situation. Your investigation traces the error back to the configuration change.
bash
# INVESTIGATION STEPS (simulated outputs)
# Step 1: Check recent deployments
kubectl rollout history deployment/cricketpulse-nginx
# Revision Change-Cause
# 1 Initial deploy
# 2 nginx-configmap: rate_limit updated (4 hours ago)
# Step 2: Check nginx configmap
kubectl get configmap nginx-configmap -o yaml | grep rate_limit
# rate_limit_requests_per_second: "5" ← was this changed from what value?
kubectl describe configmap nginx-configmap | grep -A2 "rate_limit"
# rate_limit_requests_per_second: 5
# rate_limit_burst: 10
# Comment: "reduced from 50 to 5 for testing - forgot to revert"
# ROOT CAUSE FOUND: rate limit accidentally left at 5 req/s per IP
# Normal rate was 50 req/s; 800 RPS / 5 req/s = many IPs hitting limit
# Step 3: Containment — update configmap and reload nginx
kubectl edit configmap nginx-configmap
# Change: rate_limit_requests_per_second: "5"
# To: rate_limit_requests_per_second: "50"
# Apply the change
kubectl rollout restart deployment/cricketpulse-nginx
kubectl rollout status deployment/cricketpulse-nginx --timeout=60s
# Step 4: Verify containment
# Check error rate drops within 2 minutes of nginx reload
echo "Simulated: error rate dropped from 25% to 0.1% at 22:26 UTC"
# Update incident channel:
RESOLUTION_MESSAGE = """
[RESOLVED - 22:26 UTC]
Root cause: nginx rate_limit_requests_per_second accidentally set to 5 (was 50)
Fix: reverted configmap, rolled nginx
Duration: 22 minutes (18 before detection + 4 from alert to fix)
Error budget consumed: ~2% of monthly budget
"""Step 4 — Produce the Postmortem
Using the postmortem template from Lesson 15, produce a complete postmortem document for this incident. Identify the root cause via Five Whys, list at least three action items, and identify one lesson learned from the response.
Analogy🏏Cricket
🏏 Think of it like cricket: The match doesn't truly end at the last ball — it ends when the review is written. Just as a coach, after a lost chase, sits with the footage and traces the defeat back through each decision until a systemic training gap is found, you apply the Five Whys to the 429 errors: why did requests fail, why was the rate limit set to 5, why wasn't it reverted, why wasn't the change caught — until you reach a missing review gate, not 'someone forgot'. Just as the review ends with concrete training plans, each assigned to a coach with a deadline rather than vague intentions, your postmortem lists at least three action items, each with an owner and a due date. And just as the debrief captures one thing the team did well so good habits are reinforced, you record at least one lesson learned from the response. The payoff: practising the write-up turns a one-off scare into permanent, systemic improvement — the whole point of running the drill.
python
# Complete postmortem document for the drill
# Save to: postmortems/drill-2025-06-01-score-api-429.md
# --- FILL IN THIS TEMPLATE BASED ON YOUR DRILL ---
POSTMORTEM_TEMPLATE = """
# Postmortem: Score API 429 Rate Limit Misconfiguration
## Impact
- Duration: [fill from your drill timeline]
- Severity: P2
- Users affected: [estimate based on 800 RPS * 0.25 * 22 min]
- Revenue impact: [score API is read-only — direct revenue? indirect churn?]
- SLO impact: [calculate error budget consumed]
## Timeline
[List each action with timestamp from your drill notes]
- [time] First user report via Twitter
- [time] PagerDuty alert fired
- [time] IC opened incident channel
- ...
## Root Cause (Five Whys)
Why 1: Why did users receive 429 errors?
→ nginx rate limit was set to 5 req/s per IP
Why 2: Why was it set to 5?
→ [your answer]
Why 3: Why wasn't it reverted after testing?
→ [your answer — what systemic gap allowed this]
Why 4: Why wasn't the change detected?
→ [your answer — what monitoring/review gap]
Why 5: Why did the monitoring gap exist?
→ [systemic process gap]
Root cause: [one sentence systemic statement]
## Action Items
| # | Action | Owner | Due |
|---|--------|-------|-----|
| A1 | [at least 3 action items] | | |
## Lessons Learned
What worked well: [at least 1 item]
What to improve: [at least 1 item]
"""
# Expected action items (compare after writing your own):
EXPECTED_ACTIONS = [
"A1: Add configmap change review step to deployment checklist",
"A2: Add rate limit value to Grafana dashboard for visibility",
"A3: Add nginx rate_limit_requests_per_second to change-freeze list for production",
"A4: Update runbook to include rate limit check in high-429 incident diagnosis",
]Verify Your Work
Review your drill performance against these metrics. A passing drill meets all five criteria.
Analogy🏏Cricket
🏏 Think of it like cricket: A practice session is worthless until you grade it against clear benchmarks. Just as a batting coach scores a net session on measurable targets — did the batter rotate strike, was the trigger movement quick enough, did they leave the balls outside off — you review the drill against five concrete criteria rather than a vague sense that 'it went okay'. Just as the coach times how fast the batter read the length (time to detect, target under ten minutes) and how quickly they adjusted their shot (time to contain, target under twenty), you measure detection and containment against fixed thresholds. Just as a session only passes if every benchmark is met, not most of them, a passing drill must satisfy all five: fast detection, fast containment, status page updated, a systemic root cause, and at least three owned action items. The payoff: honest scoring against a rubric shows exactly which reflex to sharpen before the real match, rather than leaving you falsely confident.
python
# Drill performance rubric
# 1. Time to detect (from symptom start to IC engaged)
# Target: < 10 minutes
# Your time: [fill from your timeline]
# 2. Time to contain (from IC engaged to error rate dropping)
# Target: < 20 minutes for P2
# Your time: [fill from your timeline]
# 3. Status page updated within 15 minutes of incident open
# Yes / No
# 4. Root cause identified via Five Whys (not stopped at 'human error')
# Yes: systemic gap identified / No: stopped at human error
# 5. Postmortem has at least 3 action items with owners and due dates
# Count: [fill]
# Scoring:
# 5/5: Outstanding — ready for on-call primary
# 4/5: Good — repeat drill with tighter time targets
# 3/5: Needs practice — run drill again with a partner as shadow
# <3: Review Lessons 13-15 and repeat drill
# Common drill mistakes to watch for:
COMMON_MISTAKES = [
"Investigating root cause before containing user impact",
"Not updating status page until fully resolved",
"Stopping at 'someone changed the config' without asking why it wasn't caught",
"Action items without owners (e.g. 'team will add monitoring')",
"Postmortem written in blame language ('engineer forgot to revert')",
]If you run this drill with a team, resist the urge to jump to the configmap fix immediately when you see the nginx configmap change in the history. Practice the discipline of the IC checklist: confirm blast radius, update status page, and assign roles before implementing any fix. Skipping these steps in drills creates muscle memory for skipping them in real incidents.
Once you're comfortable with this scenario, create your own drill scenarios based on real incidents your team has experienced. The most valuable drills practice the failures most likely to recur — look at your last 6 months of postmortem action items to identify which systemic gaps haven't been fully closed yet.
- Drill discipline: follow the IC checklist in order — do not jump to containment before opening the channel and updating the status page.
- Time-stamp every action in the drill — this produces the timeline section of the postmortem automatically.
- Root cause must be systemic: 'someone changed the config' is not a root cause; 'no review gate for configmap changes to production rate limits' is.
- Action items must have owners and due dates — 'team will do X' produces zero completed action items.
- Debrief the drill using the postmortem quality checklist from Lesson 15 before moving to the next module.