Chaos Engineering
Chaos engineering is the practice of deliberately injecting controlled failures into a system — killing servers, dropping network traffic, adding latency — to test whether it stays resilient under real-world turbulence before those…
Definition
Chaos engineering is the practice of deliberately injecting controlled failures into a system — killing servers, dropping network traffic, adding latency — to test whether it stays resilient under real-world turbulence before those failures happen unplanned.
Overview
Traditional testing checks whether a system behaves correctly under expected conditions. Chaos engineering flips that assumption: it assumes failure is inevitable in any distributed system and asks whether the system degrades gracefully or falls over completely when a dependency dies mid-request. Teams run controlled experiments — terminating instances, saturating CPU, injecting network latency, or blocking access to a database — in staging or, for mature teams, in production itself, and measure the blast radius against a steady-state hypothesis. The discipline grew out of large-scale microservices architectures, where a single service failure can cascade unpredictably through dozens of dependent services connected by a service mesh or orchestrated by Kubernetes. Chaos experiments are most valuable when paired with strong observability — without metrics, logs, and traces to show exactly how the system responded, a chaos experiment just causes an outage without producing any insight. Chaos engineering is closely tied to site reliability engineering (SRE) practice: it is one of the primary ways SRE teams validate that their error budget assumptions and incident management processes actually hold up before a real failure forces the question. Well-run chaos programs start small — a single non-critical service, during business hours, with an easy rollback — and expand scope only as confidence in the system's resilience and the team's response process grows.
Key Concepts
- Steady-state hypothesis defined before any experiment — what does 'normal' look like
- Controlled, reversible fault injection rather than uncontrolled random failure
- Blast-radius limits so experiments cannot cause unbounded customer impact
- Heavy reliance on observability tooling to interpret what happened during the experiment
- Game days — scheduled, team-wide exercises simulating major outages
- Progression from staging experiments to careful production experiments over time
- Automated chaos tooling that can run experiments continuously as part of CI/CD