In this capstone project you will apply every skill from all six modules to the CricketPulse platform. You will harden the CI/CD pipeline (Module 1–2), enforce cluster policies (Module 3), simulate and document a production incident (Module 4), run a Game Day with two fault injections (Module 5), and produce a complete Production Readiness Review (Module 6). By the end, CricketPulse will have a defensible, documented security and reliability posture across all six PRR domains.
Phase 1 — Pipeline Security (Modules 1 & 2)
Harden the CricketPulse CI/CD pipeline using the Module 2 exercise as the baseline. Verify that all five security gates are active and that the pipeline uses OIDC for all cloud authentication.
# Phase 1 verification checklist
# 1. No static credentials in any workflow file
grep -r "AWS_ACCESS_KEY_ID\|AWS_SECRET\|password" .github/workflows/ 2>/dev/null && echo "FAIL" || echo "PASS: no static credentials"
# 2. All GitHub Actions pinned to commit SHA
grep -r "uses:.*@" .github/workflows/ | grep -v "@[a-f0-9]\{40\}" && echo "WARN: unpinned actions found" || echo "PASS: all actions SHA-pinned"
# 3. Pipeline runs all five gates in parallel: dep-scan, sast, image-scan, conftest, sbom
# Verify in GitHub Actions run log: all three scan jobs show "In progress" simultaneously
# 4. Trivy blocks on HIGH/CRITICAL
trivy image cricketpulse:latest --exit-code 1 --severity HIGH,CRITICAL
echo "Trivy exit code: $?" # Expected: 0 (no HIGH/CRITICAL findings)
# 5. SBOM artifact exists for last pipeline run
gh run download --name sbom-$(git rev-parse HEAD) --dir /tmp/sbom-check
ls /tmp/sbom-check/sbom.spdx.json && echo "PASS: SBOM artifact found"
# 6. Required reviewer gate configured on production environment
gh api repos/{owner}/{repo}/environments/production | python3 -c "
import json,sys
env = json.load(sys.stdin)
reviewers = env.get('protection_rules',[])
print('PASS: required reviewers configured' if reviewers else 'FAIL: no reviewer gate')
"Phase 2 — Cluster Policy Enforcement (Module 3)
Verify all three Kyverno ClusterPolicies from the Module 3 exercise are in Enforce mode and that the CricketPulse manifests pass all policies. Run conftest against all Kubernetes manifests and confirm zero policy violations.
# Phase 2 verification
# 1. All Kyverno policies in Enforce mode
for policy in restrict-image-registries require-nonroot require-resource-limits; do
mode=$(kubectl get clusterpolicy $policy -o jsonpath='{.spec.validationFailureAction}' 2>/dev/null)
echo "$policy: ${mode:-NOT_FOUND}"
done
# Expected: all three show Enforce
# 2. CricketPulse namespace PolicyReport is clean
kubectl get policyreport -n cricketpulse -o json | python3 -c "
import json,sys
r = json.load(sys.stdin)
fails = [res for item in r.get('items',[]) for res in item.get('results',[]) if res['result']=='fail']
print(f'Policy violations: {len(fails)}')
if fails:
for f in fails: print(f' {f["policy"]}: {f["message"]}')
"
# Expected: Policy violations: 0
# 3. conftest passes on all manifests
conftest test k8s/ --policy policies/k8s/ --output table
# Expected: No failures
# 4. Kubernetes audit logging enabled
kubectl get apiserver -o yaml 2>/dev/null || cat /etc/kubernetes/audit-policy.yaml 2>/dev/null || echo "Verify audit policy exists on API server config"Phase 3 — Incident Simulation (Module 4)
Run the incident drill from Module 4's exercise. Play the role of Incident Commander for the rate-limit misconfiguration scenario. Complete the full lifecycle: detect, triage, contain, resolve, and produce a postmortem document. The postmortem must have at least three action items with owners and due dates.
# Phase 3 verification
# 1. Complete the drill (inject the fault, respond as IC)
# Inject: kubectl edit configmap nginx-configmap
# Change rate_limit_requests_per_second from 50 to 5
# Time the full response cycle:
# - T0: fault injected
# - T1: first detection (user report or alert)
# - T2: incident channel opened
# - T3: root cause identified
# - T4: fix applied (rate limit restored)
# - T5: verification complete
# Record times:
DRILL_METRICS = {
"time_to_detect": "T1 - T0", # target: <10 minutes
"time_to_contain": "T4 - T0", # target: <30 minutes
"status_page_updated": "T2 + 5 minutes", # target: within 5 min of IC engaged
}
# 2. Postmortem document produced
# File: postmortems/capstone-drill-{date}.md
# Must contain: impact, timeline, root cause (Five Whys), action items, lessons learned
# 3. Action items quality check
# Each action item must have:
# - Specific action (not "improve monitoring")
# - Owner (named individual)
# - Due date (specific date, not "soon")
# - Definition of done
echo "Phase 3 complete when: drill timed, postmortem filed, action items tracked in issue tracker"Phase 4 — Game Day (Module 5)
Run the Game Day from the Module 5 exercise. Both experiments (pod failure and payment latency) must be executed with documented hypotheses, observations, and findings. Produce the Game Day report with at least two action items.
# Phase 4: Run both experiments and produce Game Day report
# Experiment 1: Score API pod failure
# Pre-conditions: Chaos Mesh installed, CricketPulse running in staging
kubectl apply -f experiment-01-pod-failure.yaml
# Record:
# - Time to pod replacement: ______ seconds
# - Peak error rate during gap: ______ %
# - Hypothesis result: PASSED / DEVIATED
# Experiment 2: Payment service latency injection
kubectl apply -f experiment-02-payment-latency.yaml
# Record:
# - Circuit breaker trip time: ______ seconds (or did NOT trip)
# - Peak checkout error rate: ______ %
# - Hypothesis result: PASSED / DEVIATED
# Game Day report must document:
# - Steady state baseline for each experiment
# - Actual observations vs hypothesis
# - At least one action item per deviated hypothesis
# Verification:
echo "Phase 4 complete when:"
echo " - experiment-01-pod-failure.yaml exists and was applied"
echo " - experiment-02-payment-latency.yaml exists and was applied"
echo " - gamedayreports/capstone-gameday.md exists with all sections"
echo " - all chaos experiments show 'Finished' status (not still running)"
kubectl get podchaos,networkchaos -n cricketpulse
# Expected: all experiments in Finished statePhase 5 — Production Readiness Review (Module 6)
Complete the full six-domain PRR checklist for CricketPulse. Every domain must be checked and the document signed off. Any blocking items that cannot be resolved must be documented with an exception rationale, owner, and completion date.
# Phase 5: Complete PRR document
# File: docs/prr/cricketpulse-capstone.md
# Auto-verify what can be automated
echo "=== PRR Automated Verification ==="
# Security domain
echo "--- Security ---"
trivy image cricketpulse:latest --exit-code 0 --severity HIGH,CRITICAL --format json | python3 -c "import json,sys; r=json.load(sys.stdin); vulns=sum(len(x.get('Vulnerabilities',[])) for x in r.get('Results',[])); print(f'Trivy: {vulns} HIGH/CRITICAL (target: 0)')"
pip-audit --requirement requirements.txt --format json | python3 -c "import json,sys; r=json.load(sys.stdin); vulns=sum(len(d.get('vulns',[])) for d in r.get('dependencies',[])); print(f'pip-audit: {vulns} vulnerabilities (target: 0)')"
# Observability domain
echo "--- Observability ---"
curl -s "http://prometheus:9090/api/v1/rules" | python3 -c "
import json,sys
data = json.load(sys.stdin)
sli_rules = [r for g in data['data']['groups'] for r in g['rules'] if 'sli:' in r.get('name','')]
slo_alerts = [r for g in data['data']['groups'] for r in g['rules'] if 'SLOBurn' in r.get('name','')]
print(f'SLI rules: {len(sli_rules)} (target: >=1)')
print(f'SLO burn alerts: {len(slo_alerts)} (target: >=2)')
"
# Reliability domain
echo "--- Reliability ---"
kubectl get hpa -n cricketpulse
# Expected: score-api HPA with min/max configured
# Capacity domain
echo "--- Capacity ---"
echo "Load test results: check k6-results/ directory for last run"
ls k6-results/ 2>/dev/null || echo "WARN: no load test results found"
# Operations domain
echo "--- Operations ---"
echo "Runbooks present:"
ls runbooks/ 2>/dev/null | wc -l
# Compliance domain
echo "--- Compliance ---"
ls sbom*.spdx.json 2>/dev/null && echo "SBOM: found" || echo "SBOM: NOT FOUND"Final Rubric
The capstone is evaluated across four areas, each worth 25 points, for a maximum of 100 points. Each area requires a specific artifact as evidence.
# Capstone rubric
# Area 1: Pipeline Security (25 points)
# Evidence: GitHub Actions workflow file + last successful pipeline run screenshot
# Requirements:
# - 0 static credentials (5 pts)
# - All actions SHA-pinned (5 pts)
# - Parallel security gates (dep-scan, sast, image-scan) (5 pts)
# - SBOM artifact present in last pipeline run (5 pts)
# - Required reviewer gate on production environment (5 pts)
# Area 2: Cluster Policy & Compliance (25 points)
# Evidence: kubectl output + conftest output + PRR compliance section
# Requirements:
# - 3 ClusterPolicies in Enforce mode (10 pts)
# - 0 PolicyReport violations in cricketpulse namespace (5 pts)
# - conftest passes on all manifests (5 pts)
# - Audit policy covers secret access (5 pts)
# Area 3: Incident & Operations Readiness (25 points)
# Evidence: postmortem document + at least 3 runbook files
# Requirements:
# - Drill completed within time targets (MTTD <10m, MTTC <30m) (10 pts)
# - Postmortem has Five Whys root cause (5 pts)
# - 3+ action items with owners and due dates (5 pts)
# - 3+ runbooks in runbooks/ directory (5 pts)
# Area 4: Resilience & PRR (25 points)
# Evidence: Game Day report + PRR document
# Requirements:
# - Game Day report with both experiments documented (10 pts)
# - At least 2 action items from Game Day deviations (5 pts)
# - PRR document with all 6 domains checked or excepted (5 pts)
# - PRR signed off by reviewer (5 pts)
# Passing score: 70/100
# Outstanding: 90/100Do not start the capstone until all 23 preceding lessons are complete. Each phase builds on skills from its corresponding module — attempting Phase 4 (Game Day) without the Chaos Mesh skills from Lesson 18, or Phase 5 (PRR) without the SLO knowledge from Lesson 22, will produce incomplete artifacts that do not meet the rubric requirements.
The PRR document produced in Phase 5 is a reusable template for all future CricketPulse service launches. Store it in your repository under docs/prr/ and reference it in your development runbooks as the required pre-launch process. The investment in this capstone produces permanent organisational infrastructure, not just a course submission.
- Phase 1 (Security): zero static credentials, SHA-pinned actions, parallel gates, SBOM artifact, reviewer gate.
- Phase 2 (Policy): three ClusterPolicies in Enforce, zero PolicyReport violations, conftest clean.
- Phase 3 (Incident): drill MTTD <10m, MTTC <30m, postmortem with Five Whys and 3+ action items.
- Phase 4 (Game Day): both experiments documented with hypothesis vs actual, action items for any deviations.
- Phase 5 (PRR): all six domains verified, exceptions documented with owner and date, document signed off.