Chaos Engineering Cheat Sheet
Principles and tooling for deliberately injecting failure into systems to validate resilience before real incidents occur.
2 PagesAdvancedFeb 20, 2026
Principles of Chaos Engineering
Core tenets from the Principles of Chaos manifesto.
- Steady state hypothesis- Define a measurable baseline of normal system behavior before experimenting
- Vary real-world events- Inject failures that mirror real risks: server crashes, network latency, dependency outages
- Run in production (carefully)- Staging rarely reflects production traffic and topology accurately
- Minimize blast radius- Start small and scoped, with an automated abort mechanism
- Automate experiments- Run continuously, not as one-off manual exercises
LitmusChaos Pod Delete Experiment
Kubernetes-native chaos experiment manifest.
yaml
apiVersion: litmuschaos.io/v1alpha1kind: ChaosEnginemetadata: name: nginx-chaosspec: appinfo: appns: default applabel: app=nginx appkind: deployment chaosServiceAccount: litmus-admin experiments: - name: pod-delete spec: components: env: - name: TOTAL_CHAOS_DURATION value: '30' - name: CHAOS_INTERVAL value: '10' - name: FORCE value: 'false'
Toxiproxy Latency Injection
Simulate network latency between services using Toxiproxy's CLI.
bash
# Create a proxy in front of the real servicetoxiproxy-cli create -l localhost:26379 -u redis:6379 redis_proxy# Add 500ms latency with jitter to all traffictoxiproxy-cli toxic add -t latency -a latency=500 -a jitter=100 redis_proxy# Remove the toxic once the experiment is donetoxiproxy-cli toxic remove -n latency_downstream redis_proxy
Common Failure Modes to Test
Categories of faults typically injected in chaos experiments.
- Resource exhaustion- CPU, memory, or disk pressure on a node or container
- Network faults- Latency, packet loss, DNS failure, partitions between services
- Dependency failure- Downstream API returns errors or times out
- Instance termination- Kill a pod, VM, or availability zone to test failover
Chaos Mesh NetworkChaos Manifest
Injects packet loss and latency between labeled pods using Chaos Mesh's Kubernetes CRDs.
yaml
apiVersion: chaos-mesh.org/v1alpha1kind: NetworkChaosmetadata: name: checkout-network-delay namespace: chaos-testingspec: action: delay mode: fixed-percent value: '50' selector: namespaces: - production labelSelectors: app: checkout-service delay: latency: '200ms' correlation: '25' jitter: '50ms' duration: '5m' scheduler: cron: '@every 30m'
AWS Fault Injection Simulator Experiment Template
Terminates a percentage of EC2 instances in an Auto Scaling group with a CloudWatch alarm as an automatic stop condition.
json
{ "description": "Terminate 20% of checkout ASG instances", "targets": { "checkoutInstances": { "resourceType": "aws:ec2:instance", "resourceTags": { "chaos-eligible": "true" }, "selectionMode": "PERCENT(20)" } }, "actions": { "terminateInstances": { "actionId": "aws:ec2:terminate-instances", "targets": { "Instances": "checkoutInstances" } } }, "stopConditions": [ { "source": "aws:cloudwatch:alarm", "value": "arn:aws:cloudwatch:us-east-1:123456789012:alarm:checkout-error-rate-high" } ], "roleArn": "arn:aws:iam::123456789012:role/FisChaosRole"}
Chaos Engineering Maturity Levels
Typical progression as an organization matures its chaos practice.
- Level 0: Manual game days- Scheduled, human-run exercises in staging with a war room and observers
- Level 1: Scripted staging chaos- Repeatable, version-controlled experiments run on demand against non-prod
- Level 2: Scheduled production chaos- Small, blast-radius-limited experiments run automatically in prod off-peak
- Level 3: Continuous automated chaos- Experiments run constantly as part of the deploy pipeline, gating releases on resilience checks
- Level 4: Self-healing validation- Chaos verifies that automated remediation (autoscaling, circuit breakers, failover) actually fires, not just that failure occurred
Gremlin CLI Blackhole Attack
Blackholes traffic to a specific downstream host to test dependency-failure handling from the command line.
bash
gremlin attack blackhole \ --hosts checkout-worker-01,checkout-worker-02 \ --egress-ports 443 \ --hostnames payments-api.internal \ --length 120# Watch impact live, then attach the automated halt conditiongremlin attack halt --attack-id $(gremlin attack list --active --format json | jq -r '.[0].id')
Metrics That Justify a Chaos Program
What to measure to show chaos engineering is paying off, not just breaking things.
- MTTD (mean time to detect)- Should trend down as monitoring/alerting gaps found in game days get fixed
- MTTR (mean time to recover)- Should trend down as runbooks and automated failover are validated repeatedly
- Unplanned incident rate- Fewer surprises in prod as failure modes are found proactively instead
- Experiment coverage- Percentage of critical services with at least one scheduled recurring experiment
- Blast radius containment rate- Percentage of experiments that stayed within their intended scope without manual intervention
Pro Tip
Always pair a chaos experiment with an automated abort condition tied to a real SLO metric — if user impact exceeds a threshold, the experiment should halt itself immediately, not wait for a human to notice.
Was this cheat sheet helpful?
Explore Topics
#ChaosEngineering#ChaosEngineeringCheatSheet#DevOps#Advanced#PrinciplesOfChaosEngineering#LitmusChaos#Pod#Delete#CheatSheet#SkillVeris
