Lab — run a chaos experiment with AWS FIS on an EKS workload — SRE, Platform Engineering & Professional Readiness

What You'll Build

In this lab you will run a complete, production-grade chaos experiment against the MatchPulse scorecard API deployed on Amazon EKS, using AWS Fault Injection Simulator to terminate a worker node and observe the system's resilience behaviour in real time. You will establish a steady-state baseline, design and execute an FIS experiment template with stop conditions, observe the Kubernetes pod rescheduling timeline alongside SLO metric behaviour, and conclude by documenting findings in a structured experiment report.

The lab demonstrates the full chaos engineering workflow — from hypothesis to evidence-based infrastructure improvement — and surfaces a common EKS resilience gap that unit tests, integration tests, and staging tests never catch. The completed experiment report and remediation manifest are the deliverables.

Analogy🏏Cricket

🏏 Think of it like cricket: Designing a zero-downtime migration plan is like planning the changeover from a manual scoreboard to a fully digital electronic system mid-season without cancelling a single match. The old manual system (source MySQL) continues running throughout. The digital system (Aurora MySQL) is set up in parallel, receives a complete copy of all historical scorecards (full load), and then mirrors every new delivery recorded on the old board in real time (CDC). When validated, the announcement system is simply redirected to the digital board (application cutover). The rollback plan keeps the old board fully staffed and operational until the digital system is confirmed reliable — if a problem emerges, the announcement system points back at the old board within seconds.

Prerequisites

AWS CLI configured with credentials for the lab account — run `aws sts get-caller-identity` to confirm access before starting.
kubectl configured and connected to the lab EKS cluster `cricket-scorecard-lab` — run `kubectl get nodes` to confirm connectivity.
The Prometheus SLO dashboard from Lesson 6 showing live data for the MatchPulse scorecard journey — confirm all recording rule time series are visible and populated.
The FIS IAM execution role `FIS-SRE-ExperimentRole` pre-created in the lab account with permissions to terminate EC2 instances tagged as EKS worker nodes.
The CloudWatch stop-condition alarm `matchpulse-scorecard-error-rate-critical` pre-created and confirmed to be in OK state before starting the experiment.

Setup & Project Structure

The lab environment provides a three-node EKS cluster across two availability zones — two nodes in us-east-1a, one in us-east-1b — running the MatchPulse scorecard API as a three-replica Deployment. The Deployment intentionally has no PodDisruptionBudget configured. Discovering and remediating this gap is one of the lab's key learning objectives. Verify the starting state with the commands below before executing any experiment steps.

Analogy🏏Cricket

🏏 Think of it like cricket: this exercise hands you a match situation with the ground prepared and the laws written — you supply the game plan and execute it ball by ball. Just as a captain inherits the squad list, the pitch report, and the playing conditions, you inherit the source MySQL schema, sample data, and Terraform stubs for the DMS infrastructure; your job is to complete the configuration, run the SCT assessment, configure the DMS task, and work through the migration runbook step by step, the way a chase is worked over by over against a written plan. The validation script that compares row counts and checksums across the 12 tables is the third umpire of the exercise: an independent, impartial check that the new scorebook matches the old one entry for entry before anyone declares the migration complete. The payoff: practising the full sequence — assess, configure, migrate, validate — in a safe environment builds the muscle memory that makes a real zero-downtime cutover feel like a drill you've already run.

bash

# Pre-lab environment verification

# 1. Confirm EKS node topology (2 nodes in us-east-1a, 1 in us-east-1b)
kubectl get nodes -L topology.kubernetes.io/zone

# 2. Confirm MatchPulse deployment (3 Running pods)
kubectl get pods -n matchpulse -l app=scorecard-api -o wide
# Observe pod distribution — if all 3 land on 2 us-east-1a nodes,
# terminating one us-east-1a node immediately evicts all pods on that node

# 3. Confirm no PodDisruptionBudget exists (the gap you will find and fix)
kubectl get pdb -n matchpulse
# Expected: No resources found — document this as a primary finding

# 4. Confirm SLO baseline is healthy (last 30 minutes)
# Open Grafana: http://localhost:3000/d/matchpulse-slo
# Confirm matchpulse_scorecard:availability5m is stable above 0.999

# 5. Confirm CloudWatch stop-condition alarm is in OK state
aws cloudwatch describe-alarms   --alarm-names matchpulse-scorecard-error-rate-critical   --query 'MetricAlarms[0].StateValue'

Step 1 — Hypothesis and FIS Template

Before touching the FIS console, write the formal experiment hypothesis: 'When one worker node in us-east-1a is terminated, the MatchPulse scorecard API availability will remain above 99.5% and P99 latency will remain below 300ms throughout the fault injection and recovery period.' The hypothesis uses a slightly looser threshold than the production SLO (99.9%) to account for the brief pod rescheduling window — the experiment tests whether the system recovers within acceptable bounds, not whether it never degrades.

Analogy🏏Cricket

🏏 Think of it like cricket: Writing the hypothesis before creating the FIS template is like the captain writing the pre-match tactical plan before the team takes the field. The plan is not written after observing how the innings unfolds — it is written beforehand so the outcome can be objectively evaluated. If the run rate falls to 5.8 during the powerplay against a plan for 6.5, the captain does not retroactively declare the powerplay a success. Understanding why 5.8 occurred is the learning opportunity the plan was designed to produce, and it only produces that learning if the target was committed to in advance.

bash

# Create FIS experiment template via AWS CLI
CLUSTER_NAME="cricket-scorecard-lab"
FIS_ROLE_ARN=$(aws iam get-role --role-name FIS-SRE-ExperimentRole   --query 'Role.Arn' --output text)
STOP_ALARM_ARN=$(aws cloudwatch describe-alarms   --alarm-names matchpulse-scorecard-error-rate-critical   --query 'MetricAlarms[0].AlarmArn' --output text)

cat > /tmp/fis-node-termination.json << EOF
{
  "description": "Terminate 1 EKS node us-east-1a — MatchPulse scorecard resilience",
  "targets": {
    "one-node-us-east-1a": {
      "resourceType": "aws:ec2:instance",
      "resourceTags": {
        "kubernetes.io/cluster/${CLUSTER_NAME}": "owned",
        "eks.amazonaws.com/nodegroup": "scorecard-workers",
        "topology.kubernetes.io/zone": "us-east-1a"
      },
      "selectionMode": "COUNT(1)"
    }
  },
  "actions": {
    "terminate-one-node": {
      "actionId": "aws:ec2:terminate-instances",
      "targets": { "Instances": "one-node-us-east-1a" }
    }
  },
  "stopConditions": [
    { "source": "aws:cloudwatch:alarm", "value": "${STOP_ALARM_ARN}" }
  ],
  "tags": {
    "experiment-name": "node-termination-us-east-1a",
    "service": "matchpulse-scorecard",
    "blast-radius": "single-node-us-east-1a",
    "owner": "sre-lab"
  }
}
EOF

TEMPLATE_ID=$(aws fis create-experiment-template   --cli-input-json file:///tmp/fis-node-termination.json   --role-arn "${FIS_ROLE_ARN}"   --query 'experimentTemplate.id' --output text)

echo "Template created: ${TEMPLATE_ID}"

Step 2 — Execute Experiment and Observe Metrics

Start the FIS experiment and monitor three data sources simultaneously: the Kubernetes node and pod status (kubectl), the Prometheus SLO dashboard in Grafana (availability and burn rate metrics), and the FIS experiment status (AWS console or CLI). Record timestamps for each observable event: node termination confirmed, first pod eviction, last pod rescheduled and Running, availability metric recovery above hypothesis threshold. This timestamped sequence is the primary evidence for the experiment report.

Analogy🏏Cricket

🏏 Think of it like cricket: Monitoring three data sources simultaneously is like a coach simultaneously watching the batsman's footwork (Kubernetes pod status), the run rate dashboard on the dugout screen (Prometheus SLO metrics), and the match referee's signals (FIS experiment status). Each source tells a different part of the story: footwork reveals why a shot failed technically, run rate shows the scoring impact of the wicket, and the referee's signal confirms whether the dismissal was legally clean. A complete match analysis requires all three — missing any one produces an incomplete picture that may lead to the wrong corrective action.

bash

# Terminal 1: Start the FIS experiment and record start time
EXPERIMENT_ID=$(aws fis start-experiment   --experiment-template-id "${TEMPLATE_ID}"   --query 'experiment.id' --output text)
echo "Experiment started: ${EXPERIMENT_ID}  at $(date -u +%Y-%m-%dT%H:%M:%SZ)"

# Terminal 2: Watch Kubernetes node and pod status every 5 seconds
watch -n 5 'kubectl get nodes -L topology.kubernetes.io/zone &&   echo "---" &&   kubectl get pods -n matchpulse -l app=scorecard-api -o wide'

# Terminal 3: Poll FIS experiment status
watch -n 10 "aws fis get-experiment --id ${EXPERIMENT_ID}   --query '{status: experiment.state.status}'"

# Terminal 4: Watch Prometheus availability metric every 30s
watch -n 30 'curl -s "http://localhost:9090/api/v1/query?query=matchpulse_scorecard:availability5m" | python3 -c "import sys,json; d=json.load(sys.stdin); r=d["data"]["result"]; v=float(r[0]["value"][1]) if r else 0; print(f"Availability: {v:.5f} ({(1-v)*100:.3f}% errors)")"'

# ── Record these timestamps ───────────────────────────────────────────────
# T0: Experiment started
# T1: Node status changed to NotReady
# T2: First pod eviction observed
# T3: Last evicted pod reached Running on new node
# T4: Availability metric recovered above 0.995
# T5: Experiment status changed to completed

Step 3 — Document Findings and Remediate

After the experiment completes, compile your observations into the structured experiment report. The primary finding you should observe is the absence of a PodDisruptionBudget — without one, all pods on the terminated node were evicted simultaneously, leaving only one pod running during rescheduling and causing a measurable availability dip. Implement the PodDisruptionBudget fix and run the experiment a second time to validate that the remediation reduced the degradation window.

Analogy🏏Cricket

🏏 Think of it like cricket: Documenting findings and remediating is the post-innings coaching session: first, write the match report while memory is fresh, noting exactly which shots held up, which collapsed, and what field settings the opposition used; then identify the single highest-priority improvement — the technique gap that caused three wickets in quick succession — and schedule targeted coaching before the next match. Running the experiment again after remediation is the re-test net session that confirms the coaching intervention worked. If the batsman now handles the same delivery correctly, the improvement is validated. If not, the investigation continues.

bash

# Experiment report template — complete with your observations
experiment_report:
  name: "Node Termination — MatchPulse Scorecard API"
  date: "YYYY-MM-DD"
  experiment_id: ""        # fill with ${EXPERIMENT_ID}
  owner: "sre-lab"

  hypothesis:
    statement: "Availability > 99.5% and P99 latency < 300ms during node termination"
    result: "PASS | FAIL"  # fill after observing

  timeline:
    t0_experiment_started: ""
    t1_node_not_ready:      ""
    t2_first_pod_eviction:  ""
    t3_all_pods_rescheduled: ""
    t4_availability_recovered: ""

  observations:
    peak_error_rate_pct: ""
    rescheduling_latency_seconds: 0
    slo_compliance_maintained: false

  findings:
    primary: |
      No PodDisruptionBudget configured. With 2 of 3 pods on the terminated
      us-east-1a node, simultaneous eviction left only 1 pod running during
      rescheduling, producing ~60s of degraded availability.
    secondary:
      - "Pod anti-affinity not configured — suboptimal spread across nodes"
      - "Liveness probe grace period may need tuning for faster failover"

  action_items:
    - { item: "Add PodDisruptionBudget minAvailable: 2", owner: "sre-lab", due: "immediately" }
    - { item: "Add pod anti-affinity rule to spread replicas across AZs", due: "this sprint" }

---
# Apply PodDisruptionBudget remediation
cat << 'YAML' | kubectl apply -f -
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: scorecard-api-pdb
  namespace: matchpulse
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: scorecard-api
YAML

kubectl get pdb -n matchpulse
# NAME                MIN AVAILABLE   ALLOWED DISRUPTIONS
# scorecard-api-pdb   2               1

# Re-run experiment with same template to validate remediation
RETEST_ID=$(aws fis start-experiment   --experiment-template-id "${TEMPLATE_ID}"   --query 'experiment.id' --output text)
echo "Re-test experiment: ${RETEST_ID}"

Step 4 — Verify and Submit

Compare the availability metrics from the initial experiment and the re-test to validate that the PodDisruptionBudget remediation reduced or eliminated the availability degradation window. Document both experiment results in the final report, including the delta in peak error rate and rescheduling latency between the two runs. Submit the completed experiment report and the PDB manifest as the lab deliverables.

Analogy🏏Cricket

🏏 Think of it like cricket: this verification step compares two innings on the same pitch to prove the coaching change worked. Just as a team that collapsed against short bowling, then trained with a specific drill, must face the same attack again before claiming improvement, you re-run the identical chaos experiment after adding the PodDisruptionBudget and compare availability metrics between the two runs. The documented delta in peak error rate and rescheduling latency is the scorecard comparison — wickets lost and recovery time in innings one versus innings two — the only honest evidence that the remediation reduced the degradation window, rather than luck or timing. Submitting the experiment report together with the PDB manifest is filing the match report with the revised team sheet: the artefact and its evidence travel together. The payoff: you leave the lab able to prove, with before-and-after numbers, that a resilience fix actually works — the standard chaos engineering holds every remediation to.

bash

# Compare initial vs re-test experiment results using Prometheus query

# Query minimum availability during initial experiment window (use recorded T1–T4)
curl -s 'http://localhost:9090/api/v1/query_range'   --data-urlencode 'query=matchpulse_scorecard:availability5m'   --data-urlencode "start=${T1_TIMESTAMP}"   --data-urlencode "end=${T4_TIMESTAMP}"   --data-urlencode 'step=30s' | python3 -c "
import sys, json
d = json.load(sys.stdin)
values = [float(v[1]) for v in d['data']['result'][0]['values']]
print(f'Initial: min availability = {min(values):.5f}  '
      f'({(1-min(values))*100:.3f}% error rate at worst)')
"
# Re-test (substitute re-test timestamps)
# Expected after PDB: min availability > 0.995

# Final submission checklist
echo "[ ] Experiment report written with all 5 timestamps"
echo "[ ] Hypothesis result (PASS/FAIL) documented"
echo "[ ] Primary finding (missing PDB) and 2 secondary findings documented"
echo "[ ] Action items with owners and due dates"
echo "[ ] PodDisruptionBudget manifest applied (kubectl get pdb confirms)"
echo "[ ] Re-test experiment shows hypothesis now PASS"

Warning: Do not run the re-test immediately after applying the PodDisruptionBudget without confirming pod topology has actually changed. Apply the PDB, then check `kubectl get pods -n matchpulse -o wide` to verify pods are spread across nodes. If all three pods remain on the same two nodes, the PDB will prevent simultaneous eviction but rescheduling latency will still be high. Also apply the pod anti-affinity rule from the action items before the re-test to ensure pods are distributed across different AZ nodes.

Extension Challenge: Design a second experiment that injects 500ms latency on all network traffic between EKS worker nodes and the RDS database using the AWS FIS `aws:ssm:send-command` action to run a Systems Manager document configuring `tc netem` latency on the node's network interface. Observe whether the application's database query timeout is tuned correctly — if the timeout is set to 30 seconds (a common default) and the latency injection is 500ms, no errors will occur; if the timeout is 200ms, every database call will fail. Document whether your timeout configuration matches your latency SLO budget.