Incident Response & Postmortems Cheat Sheet
Structured process for detecting, responding to, and learning from production incidents, including postmortem documentation.
2 PagesIntermediateJan 25, 2026
Incident Response Roles
Common roles used during a major incident.
- Incident Commander (IC)- Owns coordination and decision-making, does not necessarily fix the issue
- Communications Lead- Manages status page updates and stakeholder communication
- Operations Lead / Subject Matter Expert- Drives technical investigation and mitigation
- Scribe- Records timeline, actions, and decisions in real time
Typical Severity Levels
Common SEV classification scheme used to prioritize response.
- SEV1 / P1- Full outage or critical data loss, all hands, immediate response
- SEV2 / P2- Major functionality degraded, significant user impact
- SEV3 / P3- Minor impact, workaround available, handled during business hours
- SEV4 / P4- Cosmetic or low-impact issue, no urgency
Postmortem Template
Minimal blameless postmortem document structure.
markdown
# Postmortem: Checkout API Outage (2026-07-08)## SummaryOne-paragraph description of impact and duration.## Impact- Duration: 42 minutes- Users affected: ~8,000- Revenue impact: estimated $X## Timeline (UTC)- 10:02 Alert fired: high 5xx rate- 10:05 IC assigned- 10:18 Root cause identified: bad config deploy- 10:44 Rollback completed, service recovered## Root CauseWhat actually broke and why.## Action Items- [ ] Add config validation to CI (owner, due date)- [ ] Add canary deploy stage (owner, due date)## Lessons LearnedWhat went well / what didn't.
Incident Response Flow
High-level sequence from detection to resolution.
- Detect- Alert fires or is reported, on-call acknowledges
- Triage- Assess severity and assign IC if warranted
- Mitigate- Stop the bleeding first (rollback, feature flag off, scale up) before root-causing
- Resolve- Confirm metrics/SLIs back to normal, close incident
- Review- Write and share the postmortem, track action items to completion
Multi-Window, Multi-Burn-Rate SLO Alert
Prometheus alerting rule that pages only when the error-budget burn rate is fast across both a short and long window, cutting false pages.
yaml
groups: - name: slo-burn-rate rules: - alert: ErrorBudgetBurnFast expr: | ( sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) ) > (14.4 * 0.001) and ( sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) ) > (14.4 * 0.001) for: 2m labels: severity: page annotations: summary: "Fast burn: budget exhausted in <2h at current rate" - alert: ErrorBudgetBurnSlow expr: | ( sum(rate(http_requests_total{status=~"5.."}[6h])) / sum(rate(http_requests_total[6h])) ) > (6 * 0.001) for: 15m labels: severity: ticket annotations: summary: "Slow burn: budget exhausted in <5d, investigate during business hours"
Postmortem Anti-Patterns
Common ways postmortems fail to produce lasting improvement.
- Naming names- Attributing root cause to a person instead of the system/process that allowed the mistake — kills future reporting honesty
- "Human error" as root cause- A stopping point, not an explanation; ask why the system made the error easy to make
- Action items with no owner or date- Untracked items rot; every action item needs a single accountable owner and due date
- Root cause singular framing- Real incidents are usually a chain of contributing factors, not one root cause — document the whole causal graph
- Writing it and shelving it- A postmortem nobody reads outside the team that wrote it doesn't spread the lesson org-wide
- Skipping near-misses- Incidents that almost happened are the cheapest data you'll ever get on a latent risk
- Closing before verification- Marking an action item done without confirming the fix actually prevents recurrence
Auto-Create Incident Channel on Page
Webhook handler that spins up a dedicated Slack channel and pins the timeline doc the moment PagerDuty fires.
javascript
app.post('/webhooks/pagerduty', async (req, res) => { const event = req.body.event; if (event.event_type !== 'incident.triggered') return res.sendStatus(200); const incident = event.data; const channelName = `inc-${incident.incident_number}-${slugify(incident.title)}`; const channel = await slack.conversations.create({ name: channelName }); await slack.conversations.setTopic({ channel: channel.channel.id, topic: `SEV${incident.urgency === 'high' ? '1' : '3'} | ${incident.html_url}`, }); await slack.chat.postMessage({ channel: channel.channel.id, text: `:rotating_light: *${incident.title}*\nIC: unassigned — react with :raised_hand: to claim\nTimeline doc: ${await createTimelineDoc(incident)}`, }); res.sendStatus(200);});
Cognitive Biases That Distort Investigations
Biases to actively guard against while reconstructing an incident timeline.
- Hindsight bias- "It was obvious in retrospect" — it wasn't, given what responders knew at the time; reconstruct their actual view
- Outcome bias- Judging a decision by how it turned out rather than whether it was reasonable given available information
- Confirmation bias- Latching onto the first plausible cause and only seeking evidence that supports it
- Fundamental attribution error- Blaming an individual's carelessness while ignoring situational/system factors that set them up to fail
- Normalization of deviance- A known risk that hasn't caused an incident yet gets quietly accepted as "fine"
Automated Status Page Update
CLI-friendly script to post an incident status update via the Statuspage API during an active SEV.
bash
curl -s -X PATCH \ "https://api.statuspage.io/v1/pages/${PAGE_ID}/incidents/${INCIDENT_ID}" \ -H "Authorization: OAuth ${STATUSPAGE_API_KEY}" \ -d 'incident[status]=monitoring' \ -d 'incident[body]=A fix has been deployed. We are monitoring the results.' \ -d 'incident[component_ids][]=comp_checkout_api' \ -d 'incident[components][comp_checkout_api]=degraded_performance'
Pro Tip
Mitigate first, diagnose second — rolling back a bad deploy takes minutes and restores service, while root-causing under pressure can extend an outage unnecessarily.
Was this cheat sheet helpful?
Explore Topics
#IncidentResponsePostmortems#IncidentResponsePostmortemsCheatSheet#DevOps#Intermediate#IncidentResponseRoles#TypicalSeverityLevels#PostmortemTemplate#IncidentResponseFlow#CheatSheet#SkillVeris
