Lab — Observability, Security Hardening and On-Call Runbook

What You'll Build

With the core infrastructure provisioned in M6 Lesson 3, you will now add the operational layer that separates a toy deployment from a production system: CloudWatch dashboards showing API health, ALB latency and error rates, and RDS connection counts; CloudWatch alarms with SNS notifications for p99 latency over 500ms and 5xx error rate over 1%; VPC Flow Log analysis for security anomaly detection; IAM Access Analyzer to verify least-privilege posture; and a written runbook that an on-call engineer can follow at 2am during an incident. Production infrastructure without observability is an aircraft without instruments — technically flying but dangerous.

Analogy🏏Cricket

🏏 Think of it like cricket: This script is like the pre-match ground inspection conducted by the match referee, pitch curator, and captains before a Test match begins. Before Rohit Sharma and the opposition captain walk out for the toss, the curator has measured pitch moisture, taken grass length readings, and documented the surface condition — creating a baseline against which any afternoon deterioration can be measured.Just as this structured inspection prevents surprises and creates a documented record, the inventory script creates a documented baseline for a server against which future anomalies can be compared. Just as a ground inspection without a checklist might miss a drainage issue that affects the afternoon session, a server assessment without a structured script might miss a nearly-full disk that causes a midnight deployment failure.The insight is that the value of a structured inspection is not just the current findings but the reproducible method — the same script run tomorrow highlights exactly what changed.

Prerequisites

M6 Lesson 3 infrastructure deployed — VPC, ALB, ASG, RDS and Lambda running
An email address to receive SNS alarm notifications
AWS CLI configured with CloudWatch, SNS and IAM Access Analyzer permissions

Step 1 — CloudWatch Dashboard and Alarms

Create the CloudWatch operational dashboard and configure alarms for the critical SLO metrics. The dashboard shows the four golden signals: latency (ALB p99), traffic (ALB request count), errors (ALB 5xx rate) and saturation (RDS CPU, connection count). Alarms alert the on-call engineer when SLOs are breached.

bash

#!/bin/bash
# CloudWatch dashboard and alarms for cricket analytics platform
set -euo pipefail

REGION='ap-south-1'
ALB_ARN=$(aws elbv2 describe-load-balancers \
    --query 'LoadBalancers[?contains(LoadBalancerName,`cricket`)].LoadBalancerArn' \
    --output text | head -1)
ALB_SUFFIX=$(echo "$ALB_ARN" | sed 's|.*loadbalancer/||')

# SNS topic for alarm notifications
SNS_ARN=$(aws sns create-topic --name cricket-alarms --query TopicArn --output text)
read -r -p 'Enter email for alarm notifications: ' EMAIL
aws sns subscribe --topic-arn "$SNS_ARN" --protocol email \
    --notification-endpoint "$EMAIL"
echo "SNS configured — confirm subscription email"

echo
echo '=== CloudWatch Dashboard ==='
aws cloudwatch put-dashboard \
    --dashboard-name 'CricketAnalyticsPlatform' \
    --dashboard-body "$(cat << DASHBOARD
{
  \"widgets\": [
    {
      \"type\": \"metric\",
      \"x\": 0, \"y\": 0, \"width\": 12, \"height\": 6,
      \"properties\": {
        \"title\": \"API Latency p99 (target: <500ms)\",
        \"metrics\": [[\"AWS/ApplicationELB\",\"TargetResponseTime\",\"LoadBalancer\",\"${ALB_SUFFIX}\",{\"stat\":\"p99\",\"label\":\"p99 latency\"}]],
        \"period\": 60, \"view\": \"timeSeries\"
      }
    },
    {
      \"type\": \"metric\",
      \"x\": 12, \"y\": 0, \"width\": 12, \"height\": 6,
      \"properties\": {
        \"title\": \"5xx Error Rate (target: <1%)\",
        \"metrics\": [
          [\"AWS/ApplicationELB\",\"HTTPCode_Target_5XX_Count\",\"LoadBalancer\",\"${ALB_SUFFIX}\",{\"stat\":\"Sum\"}],
          [\".\",\"RequestCount\",\".\",\".\",{\"stat\":\"Sum\"}]
        ],
        \"period\": 60
      }
    },
    {
      \"type\": \"metric\",
      \"x\": 0, \"y\": 6, \"width\": 12, \"height\": 6,
      \"properties\": {
        \"title\": \"Request Rate (req/min)\",
        \"metrics\": [[\"AWS/ApplicationELB\",\"RequestCount\",\"LoadBalancer\",\"${ALB_SUFFIX}\",{\"stat\":\"Sum\",\"period\":60}]]
      }
    },
    {
      \"type\": \"metric\",
      \"x\": 12, \"y\": 6, \"width\": 12, \"height\": 6,
      \"properties\": {
        \"title\": \"RDS CPU Utilisation\",
        \"metrics\": [[\"AWS/RDS\",\"CPUUtilization\",\"DBInstanceIdentifier\",\"cricket-analytics-production\",{\"stat\":\"Average\"}]]
      }
    }
  ]
}
DASHBOARD
)"
echo 'Dashboard created: https://console.aws.amazon.com/cloudwatch/home#dashboards:name=CricketAnalyticsPlatform'

echo
echo '=== CloudWatch Alarms ==='

# Alarm 1: p99 latency > 500ms
aws cloudwatch put-metric-alarm \
    --alarm-name 'cricket-api-high-latency' \
    --alarm-description 'API p99 latency > 500ms for 3 consecutive minutes' \
    --namespace 'AWS/ApplicationELB' \
    --metric-name 'TargetResponseTime' \
    --dimensions Name=LoadBalancer,Value="$ALB_SUFFIX" \
    --statistic 'p99' \
    --period 60 \
    --evaluation-periods 3 \
    --threshold 0.5 \
    --comparison-operator 'GreaterThanThreshold' \
    --alarm-actions "$SNS_ARN" \
    --treat-missing-data 'notBreaching'

# Alarm 2: 5xx error rate > 1%
aws cloudwatch put-metric-alarm \
    --alarm-name 'cricket-api-high-errors' \
    --alarm-description '5xx error rate > 1% for 2 consecutive minutes' \
    --namespace 'AWS/ApplicationELB' \
    --metric-name 'HTTPCode_Target_5XX_Count' \
    --dimensions Name=LoadBalancer,Value="$ALB_SUFFIX" \
    --statistic 'Sum' \
    --period 60 \
    --evaluation-periods 2 \
    --threshold 100 \
    --comparison-operator 'GreaterThanThreshold' \
    --alarm-actions "$SNS_ARN" \
    --treat-missing-data 'notBreaching'

# Alarm 3: RDS CPU > 80%
aws cloudwatch put-metric-alarm \
    --alarm-name 'cricket-rds-high-cpu' \
    --alarm-description 'RDS CPU > 80% for 5 consecutive minutes' \
    --namespace 'AWS/RDS' \
    --metric-name 'CPUUtilization' \
    --dimensions Name=DBInstanceIdentifier,Value=cricket-analytics-production \
    --statistic 'Average' \
    --period 60 \
    --evaluation-periods 5 \
    --threshold 80 \
    --comparison-operator 'GreaterThanThreshold' \
    --alarm-actions "$SNS_ARN"

echo 'Alarms created'

Step 2 — Security Hardening and IAM Audit

Run automated security checks using IAM Access Analyzer, AWS Security Hub and a custom bash audit script. Document any findings and remediate before declaring the platform production-ready. Security hardening is not a one-time activity — it should be included in the definition of done for every deployment.

bash

#!/bin/bash
# Security hardening audit — cricket analytics platform

echo '=== Security Audit: Cricket Analytics Platform ==='
PASS=0; FAIL=0
check() {
    local desc="$1" cmd="$2" expect="$3"
    printf '%-55s ' "${desc}:"
    result=$(eval "$cmd" 2>/dev/null || echo 'ERROR')
    if echo "$result" | grep -qE "$expect"; then
        echo 'PASS'; PASS=$((PASS+1))
    else
        echo "FAIL ($result)"; FAIL=$((FAIL+1))
    fi
}

# IAM checks
check 'Root access key absent' \
    'aws iam get-account-summary --query SummaryMap.AccountAccessKeysPresent --output text' \
    '^0$'

check 'Root MFA enabled' \
    'aws iam get-account-summary --query SummaryMap.AccountMFAEnabled --output text' \
    '^1$'

check 'CloudTrail enabled' \
    'aws cloudtrail describe-trails --query "trailList[?IsMultiRegionTrail].Name" --output text' \
    '.+'

# S3 checks
check 'S3 Block Public Access at account level' \
    'aws s3control get-public-access-block --account-id $(aws sts get-caller-identity --query Account --output text) --query PublicAccessBlockConfiguration.BlockPublicAcls --output text' \
    '^True$'

# VPC checks
check 'VPC Flow Logs enabled' \
    'aws ec2 describe-flow-logs --filter Name=resource-type,Values=VPC --query "length(FlowLogs)" --output text' \
    '^[1-9]'

check 'Default VPC deleted' \
    'aws ec2 describe-vpcs --filters Name=isDefault,Values=true --query "length(Vpcs)" --output text' \
    '^0$'

# RDS checks
check 'RDS encryption enabled' \
    'aws rds describe-db-instances --query "DBInstances[?DBInstanceIdentifier==\`cricket-analytics-production\`].StorageEncrypted" --output text' \
    '^True$'

check 'RDS Multi-AZ enabled' \
    'aws rds describe-db-instances --query "DBInstances[?DBInstanceIdentifier==\`cricket-analytics-production\`].MultiAZ" --output text' \
    '^True$'

check 'RDS deletion protection on' \
    'aws rds describe-db-instances --query "DBInstances[?DBInstanceIdentifier==\`cricket-analytics-production\`].DeletionProtection" --output text' \
    '^True$'

echo
echo "Results: ${PASS} PASSED, ${FAIL} FAILED"
[[ $FAIL -gt 0 ]] && echo 'Remediate FAILED checks before declaring production-ready'

echo
echo '=== IAM Access Analyzer — check for public access ==='
ANALYZER_ARN=$(aws accessanalyzer list-analyzers \
    --query 'analyzers[0].arn' --output text 2>/dev/null)
if [[ -n "$ANALYZER_ARN" && "$ANALYZER_ARN" != 'None' ]]; then
    aws accessanalyzer list-findings \
        --analyzer-arn "$ANALYZER_ARN" \
        --filter '{"status":{"eq":["ACTIVE"]}}' \
        --query 'findings[*].{Type:findingType, Resource:resource, Status:status}' \
        --output table
else
    echo 'Create an IAM Access Analyzer in the AWS console to detect public access findings'
fi

Step 3 — On-Call Runbook

Write the operational runbook for the cricket analytics platform. A runbook is the on-call engineer's guide for responding to incidents — it should be clear enough to follow at 2am under pressure, without requiring deep knowledge of the system. Each procedure should include what to check, what to do and how to verify the fix worked.

bash

# Write the runbook to a Markdown file
cat > ~/cricket_capstone/RUNBOOK.md << 'RUNBOOK'
# Cricket Analytics Platform — On-Call Runbook

## Alarm Severity Levels
- P1 (page immediately): All users affected, data loss risk
- P2 (respond within 30 min): Partial outage or degraded performance
- P3 (business hours): Non-critical, investigate when available

## Architecture Quick Reference
- API entry: Route 53 → ALB → EC2 ASG (private subnet ap-south-1a/b/c)
- Databases: DynamoDB (live scores) + RDS PostgreSQL Multi-AZ (analytics)
- Ingestion: S3 upload → Lambda trigger → DynamoDB + RDS write
- DNS failover: Route 53 health check → automatic failover to ap-southeast-1
- Dashboard: https://console.aws.amazon.com/cloudwatch/home#dashboards:name=CricketAnalyticsPlatform

---

## Procedure 1: High API Latency (Alarm: cricket-api-high-latency)
**Severity:** P2 if p99 > 500ms; P1 if p99 > 2000ms

### Diagnosis
```bash
# Check ALB target health
aws elbv2 describe-target-health --target-group-arn TG_ARN \
    --query 'TargetHealthDescriptions[*].{ID:Target.Id,Health:TargetHealth.State}'

# Check EC2 CPU utilisation (high CPU = need more instances)
aws cloudwatch get-metric-statistics \
    --namespace AWS/EC2 --metric-name CPUUtilization \
    --dimensions Name=AutoScalingGroupName,Value=cricket-analytics-asg \
    --start-time $(date -d '15 minutes ago' -Iseconds) \
    --end-time $(date -Iseconds) --period 60 --statistics Average

# Check RDS connections and CPU (DB bottleneck)
aws cloudwatch get-metric-statistics \
    --namespace AWS/RDS --metric-name DatabaseConnections \
    --dimensions Name=DBInstanceIdentifier,Value=cricket-analytics-production \
    --start-time $(date -d '15 minutes ago' -Iseconds) \
    --end-time $(date -Iseconds) --period 60 --statistics Average
```

### Resolution
```bash
# If EC2 CPU > 70%: manually increase desired capacity
aws autoscaling set-desired-capacity \
    --auto-scaling-group-name cricket-analytics-asg \
    --desired-capacity 8

# If RDS connections > 400: restart application servers to release connections
aws autoscaling start-instance-refresh \
    --auto-scaling-group-name cricket-analytics-asg \
    --preferences '{"MinHealthyPercentage": 80}'
```
### Verify: p99 latency drops below 500ms within 5 minutes

---

## Procedure 2: RDS Failover (primary AZ failure)
**Severity:** P1 — automatic, but requires monitoring

### What happens automatically
1. RDS detects primary failure (60-120 seconds)
2. DNS for the RDS endpoint updates to standby IP
3. Application reconnects using the same endpoint hostname

### Engineer actions
```bash
# Monitor failover progress
aws rds describe-db-instances \
    --db-instance-identifier cricket-analytics-production \
    --query 'DBInstances[0].{Status:DBInstanceStatus,AZ:AvailabilityZone,MultiAZ:MultiAZ}'

# Verify application reconnection (check ALB 5xx rate)
aws cloudwatch get-metric-statistics \
    --namespace AWS/ApplicationELB --metric-name HTTPCode_Target_5XX_Count \
    --start-time $(date -d '5 minutes ago' -Iseconds) \
    --end-time $(date -Iseconds) --period 60 --statistics Sum
```
### Verify: API 5xx rate returns to baseline within 3 minutes of failover

---

## Procedure 3: Deploy New Application Version
**Severity:** N/A (planned)

```bash
# 1. Update AMI in Terraform and apply
cd ~/cricket_capstone/environments/production
# Edit modules/compute/main.tf: update ami_id variable
terraform plan -out=deploy.tfplan
terraform apply deploy.tfplan

# 2. Trigger instance refresh (zero-downtime rolling deployment)
aws autoscaling start-instance-refresh \
    --auto-scaling-group-name cricket-analytics-asg \
    --preferences '{"MinHealthyPercentage": 80, "InstanceWarmup": 60}'

# 3. Monitor refresh status
aws autoscaling describe-instance-refreshes \
    --auto-scaling-group-name cricket-analytics-asg \
    --query 'InstanceRefreshes[0].{Status:Status,Percentage:PercentageComplete}'
```
### Verify: All instances show new AMI ID; p99 latency unchanged; zero 5xx errors

---

## Procedure 4: RDS Point-in-Time Recovery
**Severity:** P1 — data loss scenario

```bash
# Restore to specific point in time (use time BEFORE data was deleted/corrupted)
aws rds restore-db-instance-to-point-in-time \
    --source-db-instance-identifier cricket-analytics-production \
    --target-db-instance-identifier cricket-analytics-recovered \
    --restore-time 2024-04-15T14:00:00Z \
    --db-instance-class db.t3.medium \
    --multi-az

# Wait for recovery (15-30 minutes)
aws rds wait db-instance-available \
    --db-instance-identifier cricket-analytics-recovered

# Validate data, then update application to use recovered instance
# NEVER delete the original until data integrity is confirmed
```

## Contact Escalation
- Primary on-call: Check PagerDuty schedule
- Database issues: @database-team Slack channel
- Network/VPC issues: @platform-team Slack channel
- AWS Support: https://console.aws.amazon.com/support (Business Support plan)
RUNBOOK

echo 'Runbook written to ~/cricket_capstone/RUNBOOK.md'

Pro Tip

The four golden signals of SRE observability are latency (how long requests take), traffic (how many requests are being processed), errors (the rate of requests that fail) and saturation (how full the service's capacity is). Configure CloudWatch alarms on all four signals. Latency and errors catch user-facing impact; traffic catches unexpected load patterns; saturation (CPU, memory, connection pool fullness) catches capacity issues before they become latency and error problems. Instrument each signal independently so you can diagnose which dimension is causing an incident.

The four golden signals (latency, traffic, errors, saturation) are the minimum observable set for any production service — configure CloudWatch alarms on all four before declaring a service production-ready.
Alarm evaluation periods should match the incident response time — a 1-minute evaluation with 3 consecutive periods means 3 minutes to trigger, giving time for transient spikes to pass before paging.
The on-call runbook must be written and tested before an incident — a runbook that has never been followed will have gaps discovered at 2am under pressure.
IAM Access Analyzer finds resources with public access that should be private — run it after every deployment and remediate active findings within 24 hours.
Security hardening is an automated, repeatable process — the audit script in this lab should be added to the CI/CD pipeline to run on every deployment and fail if security checks regress.
RDS point-in-time recovery restores to a new instance — validate data integrity on the recovered instance before updating application connection strings; never delete the original until recovery is confirmed complete.