100% Free Forever
AI-Powered Learning
Industry Expert Content
Certificates & Badges
Learn At Your Own Pace
SRE, Platform Engineering & Professional Readiness
30 minadvanced

Chaos engineering — principles, blast radius control and GameDays

Production systems fail in ways engineers do not anticipate during design. Dependencies go unreachable, disks fill silently, network packets experience latency spikes at statistically unlikely moments, and cascading failures emerge from interactions between components that are individually well-behaved. Conventional responses — writing more tests, adding more monitoring — address known failure modes but leave unknown ones undiscovered until a production incident reveals them.

Chaos engineering addresses this gap by deliberately injecting controlled failures into production or production-equivalent environments and observing whether the system behaves according to its steady-state hypothesis. The key word is 'controlled': chaos engineering is not random destruction but a scientific experimental process with defined hypotheses, blast radius constraints, automatic rollback conditions, and formal result documentation.

Practised correctly, it transforms the question from 'is our system resilient?' — which can only be answered after an uncontrolled incident — to 'in what specific ways is our system resilient, and under what conditions does that resilience break down?'

Analogy🏏Cricket
🏏 Think of it like cricket: The SLI/SLO/SLA framework maps directly onto how a cricket franchise manages player performance. The SLI is the raw measurement from the scorecard: batting average, strike rate, economy rate — observable, precise, agreed-upon numbers. The SLO is the coaching staff's internal target: 'our number-three batsman should maintain a minimum strike rate of 130 in T20 matches.' The SLA is the franchise's contract with the board: 'if the squad averages below 7.0 runs per over across 20 matches, the franchise pays a performance penalty.' Just as the coaching staff monitors the SLO target daily to intervene before the SLA consequence triggers, SRE teams monitor SLOs to act before contractual SLAs are breached. The insight: SLIs, SLOs, and SLAs are not bureaucratic formalities — they are the measurement infrastructure that makes reliability a manageable engineering problem.
Lesson 4 of 40
0% complete