Site Reliability Engineering (SRE) Basics Cheat Sheet
Foundational SRE concepts including SLIs, SLOs, error budgets, and toil reduction for running reliable production systems.
Core SRE Vocabulary
The key definitions underpinning SRE practice.
- SLI (Service Level Indicator)- A quantitative measure of a service's behavior, e.g. request latency, error rate
- SLO (Service Level Objective)- A target value or range for an SLI over a time window, e.g. 99.9% of requests < 300ms
- SLA (Service Level Agreement)- A contractual promise to a customer, usually with consequences if the SLO is missed
- Error Budget- The allowed amount of unreliability (1 - SLO) within a window, used to balance velocity vs stability
- Toil- Manual, repetitive, automatable operational work that scales linearly with service growth
Error Budget Calculation
Compute remaining error budget for a 99.9% SLO over 30 days.
slo_target = 0.999total_minutes = 30 * 24 * 60 # 43200allowed_downtime_minutes = total_minutes * (1 - slo_target)print(allowed_downtime_minutes) # 43.2 minutes/month# If 12 minutes of downtime already occurred this month:remaining_budget = allowed_downtime_minutes - 12print(remaining_budget) # 31.2 minutes remaining
Multi-Window Burn Rate Alert
Example Prometheus alert firing on fast error-budget burn.
- alert: HighErrorBudgetBurn expr: | ( sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) ) > (14.4 * 0.001) for: 2m labels: severity: page annotations: summary: "Burning error budget 14.4x faster than sustainable"
Key SRE Practices
Recurring activities that operationalize reliability.
- Blameless postmortems- Root-cause analysis focused on systems and process, not individuals
- Capacity planning- Forecasting resource needs ahead of demand growth
- Release engineering- Standardized, repeatable build and deployment processes
- On-call rotation- Shared, time-boxed responsibility for responding to production alerts
SLO as Code (OpenSLO)
Define an SLO declaratively so it's version-controlled and can be rendered into alerts by tooling like Sloth.
apiVersion: openslo/v1kind: SLOmetadata: name: checkout-api-availabilityspec: service: checkout-api description: 99.9% of checkout requests succeed over a rolling 30d window indicator: thresholdMetric: metricSource: type: prometheus spec: query: | sum(rate(http_requests_total{job="checkout-api",status!~"5.."}[5m])) / sum(rate(http_requests_total{job="checkout-api"}[5m])) timeWindow: - duration: 30d isRolling: true budgetingMethod: Occurrences objectives: - displayName: Availability target: 0.999
Paging Severity Levels
Standard tiers for triaging incoming alerts and deciding who they wake up.
- SEV1 / P1- Full outage or major customer impact; page immediately, open an all-hands incident
- SEV2 / P2- Partial degradation or single-region impact; page on-call, no all-hands required
- SEV3 / P3- Minor impact or a workaround exists; file a ticket, business-hours response
- SEV4 / P4- Cosmetic or low-risk; backlog, no paging
- Escalation policy- If primary on-call doesn't acknowledge within N minutes, auto-escalate to secondary, then manager
Multi-Window, Multi-Burn-Rate Thresholds
Standard Google SRE burn-rate table for a 99.9% SLO over 30 days, combining short and long windows to catch both fast and slow burns.
# For a 99.9% SLO (0.1% error budget) over a 30-day window:## Burn rate | Long window | Short window | Budget consumed | Response# 14.4x | 1h | 5m | 2% in 1h | page (fast burn)# 6x | 6h | 30m | 5% in 6h | page (fast burn)# 3x | 24h | 2h | 10% in 24h | ticket (slow burn)# 1x | 72h | 6h | 10% in 72h | ticket (slow burn)## burn_rate = (1 - SLI) / (1 - SLO)# Require BOTH the long-window and short-window condition to be true# so you don't page on a brief blip that self-resolves before the# short window closes.
Toil Reduction Checklist
Questions to ask before accepting a piece of operational work as recurring toil.
- Manual- Does it require a human to execute each time, with no automation?
- Repetitive- Has it been done more than twice in a similar way?
- Automatable- Could a script or tool do this with the same or better quality?
- No enduring value- Does completing it leave the system no better than before it was done?
- Scales with growth- Does the effort grow linearly, or worse, with traffic or service count?
- If 3+ are yes- It's toil — track hours spent and cap team toil at under 50% of on-call time, per Google's guideline
Production Readiness Review Checklist
Gate a new service against this before it carries production traffic and inherits an on-call rotation.
production_readiness_review: observability: - dashboards published for the four golden signals - alerts wired to on-call, tested via synthetic firing - distributed tracing enabled with correlation IDs reliability: - SLOs defined and reviewed with stakeholders - dependency failure modes documented: timeouts, retries, circuit breakers - load tested to 2x expected peak traffic operability: - runbook exists for the top 5 likely failure scenarios - rollback procedure tested, not just documented - capacity plan covers the next 2 quarters of growth ownership: - on-call rotation staffed with an escalation policy - service has a named owning team in the catalog
When the error budget is exhausted, freeze risky feature launches and redirect the team to reliability work — this is the mechanism that actually enforces the SLO, not the number itself.