What You'll Build
This capstone lab is the production readiness audit and comprehensive verification of the complete IPL analytics platform that has been built, hardened, and instrumented across M1 through M6. Rather than building new features, you will conduct a systematic audit across six dimensions — supply chain security, workload security, network security, storage and data durability, observability and alerting, and deployment automation — producing a scored production readiness report that identifies any remaining gaps between the platform's current state and the production standard established across the course. Each dimension is assessed against a checklist derived from the lessons, with each item having a verifiable test command that produces a binary pass or fail. The final report is the capstone artefact: a complete picture of a production-grade containerised platform from the container image layer through every Kubernetes abstraction layer to the external load balancer and certificate management system.
The capstone's secondary objective is to trace every design decision made across the course back to the principle it implements: the `readOnlyRootFilesystem: true` in the Deployment spec traces to M1 Lesson 1's OverlayFS discussion, the `WaitForFirstConsumer` StorageClass traces to M4 Lesson 25's AZ-aware provisioning, the `condition: service_healthy` dependency traces to M2 Lesson 8's Compose healthcheck discussion, now expressed as an init container in Kubernetes. Making these connections explicit is the capstone's learning objective: demonstrating that the complete platform is a coherent architectural expression of principles learned throughout the course, not a collection of independently applied configurations.
Production Readiness Dimensions
The six audit dimensions map onto the six modules of the course: supply chain security covers the M1 and M2 build, scan, and signing pipeline; workload security covers M3 and M5 RBAC, PSA, and securityContext requirements; network security covers M3 Service design and M5 NetworkPolicies; storage and data durability covers M4 StatefulSets, StorageClass, and Velero backup; observability covers M6 metrics, traces, alerts, and dashboards; and deployment automation covers M6 GitOps with Argo CD and the complete image-to-production pipeline. Each dimension has five checklist items worth two points each, for a total of 60 points. A score of 50+ is production-ready; 40-49 is ready with minor gaps; below 40 requires addressing critical gaps before production deployment.
- Dimension 1 — Supply Chain Security: image built with multi-stage Dockerfile on scratch/distroless, Trivy scan with zero CRITICAL/HIGH CVEs, Cosign signature with Rekor log entry, SBOM attached as OCI artifact, Kyverno policy enforcing approved registry.
- Dimension 2 — Workload Security: all Pods compliant with PSA restricted profile, dedicated ServiceAccounts with automountServiceAccountToken false, securityContext with readOnlyRootFilesystem/runAsNonRoot/capabilities.drop ALL, IRSA for cloud access, resource limits on every container.
- Dimension 3 — Network Security: default-deny-all NetworkPolicy in production namespace, DNS egress allow-rule present, service-specific ingress/egress allow-rules for each workload, no database ports published with -p, Ingress with TLS and cert-manager auto-renewal.
- Dimension 4 — Storage and Data Durability: StatefulSet with volumeClaimTemplate using Retain reclaim policy, gp3-encrypted StorageClass with WaitForFirstConsumer, Velero scheduled backup with tested restore, VolumeSnapshot nightly schedule, no reclaimPolicy Delete on production PVCs.
- Dimension 5 — Observability and Alerting: Prometheus scraping all services with ServiceMonitor, PrometheusRule with symptom-based alerts (error rate, p99 latency), distributed traces in Jaeger with cross-service context propagation, Grafana dashboards with RED metrics, Alertmanager routing to PagerDuty/Slack.
- Dimension 6 — Deployment Automation: Argo CD Application in Synced+Healthy state, selfHeal and prune enabled, image update via CI manifest commit (no direct kubectl apply), Sealed Secrets or External Secrets for credential management, GitOps drift detection verified.
Setup & Project Structure
Before running the audit, confirm the complete platform is deployed and all services are healthy. The audit requires a running platform — a stopped or partially deployed platform will produce false failures that cannot be distinguished from genuine configuration gaps. Start the audit only when all Pods in the production namespace show Ready status.
# Pre-audit: confirm complete platform is running
echo "=== Pre-Audit Platform Health Check ==="
echo -n "All production Pods ready: "
NOT_READY=$(kubectl get pods -n production --field-selector=status.phase!=Running -o name 2>/dev/null | wc -l | tr -d ' ')
[ "$NOT_READY" -eq "0" ] && echo "PASS ✓" || echo "FAIL ✗ ($NOT_READY not running)"
echo -n "Argo CD Application Synced: "
argocd app get ipl-platform -o json 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); print('PASS ✓' if d['status']['sync']['status']=='Synced' else 'FAIL ✗')"
echo -n "Monitoring stack running: "
PROM_READY=$(kubectl get pods -n monitoring -l app.kubernetes.io/name=prometheus --field-selector=status.phase=Running -o name | wc -l | tr -d ' ')
[ "$PROM_READY" -gt "0" ] && echo "PASS ✓" || echo "FAIL ✗"
echo -n "API endpoint responding: "
kubectl port-forward service/ipl-api 8080:8000 -n production &
PF_PID=$!; sleep 3
STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health 2>/dev/null)
kill $PF_PID 2>/dev/null
[ "$STATUS" = "200" ] && echo "PASS ✓" || echo "FAIL ✗ (HTTP $STATUS)"
echo ""
echo "Platform health confirmed. Beginning production readiness audit." Step 1 — Supply Chain and Workload Security Audit
# Dimensions 1 and 2: Supply Chain + Workload Security
SCORE=0; MAX=20
pass() { echo " PASS ✓ $1"; ((SCORE+=2)); }
fail() { echo " FAIL ✗ $1"; }
echo "=== DIMENSION 1: Supply Chain Security (10 points) ==="
# 1.1 Multi-stage Dockerfile producing non-root image
echo -n "1.1 Non-root user in image: "
RUNUSER=$(docker inspect 123456789.dkr.ecr.ap-south-1.amazonaws.com/ipl-scorecard:latest --format "{{.Config.User}}" 2>/dev/null)
[ "$RUNUSER" != "root" ] && [ -n "$RUNUSER" ] && pass "user=$RUNUSER" || fail "image runs as root"
# 1.2 Trivy scan: zero CRITICAL/HIGH CVEs
echo -n "1.2 Trivy: zero CRITICAL/HIGH: "
trivy image --severity CRITICAL,HIGH --exit-code 1 --quiet 123456789.dkr.ecr.ap-south-1.amazonaws.com/ipl-scorecard:latest > /dev/null 2>&1 && pass "" || fail ""
# 1.3 Cosign signature present
echo -n "1.3 Cosign signature valid: "
cosign verify --certificate-identity-regexp "https://github.com/org/ipl-scorecard.*" --certificate-oidc-issuer "https://token.actions.githubusercontent.com" 123456789.dkr.ecr.ap-south-1.amazonaws.com/ipl-scorecard:latest > /dev/null 2>&1 && pass "" || fail ""
# 1.4 SBOM attached as OCI artifact
echo -n "1.4 SBOM attached to image: "
cosign tree 123456789.dkr.ecr.ap-south-1.amazonaws.com/ipl-scorecard:latest 2>/dev/null | grep -q "sbom" && pass "" || fail ""
# 1.5 Kyverno approved-registry policy in Enforce mode
echo -n "1.5 Registry policy enforced: "
kubectl get clusterpolicy require-approved-registry -o jsonpath='{.spec.validationFailureAction}' 2>/dev/null | grep -q "Enforce" && pass "" || fail ""
echo ""
echo "=== DIMENSION 2: Workload Security (10 points) ==="
# 2.1 PSA restricted enforcement
echo -n "2.1 PSA restricted enforced: "
PSA=$(kubectl get ns production -o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce}' 2>/dev/null)
[ "$PSA" = "restricted" ] && pass "" || fail "PSA enforce=$PSA"
# 2.2 All API Pods run as non-root
echo -n "2.2 API Pods non-root: "
ROOT_PODS=$(kubectl get pods -l app=ipl-api -n production -o json | python3 -c "import sys,json; pods=json.load(sys.stdin)['items'];
bad=[p['metadata']['name'] for p in pods
if not p['spec'].get('securityContext',{}).get('runAsNonRoot')
and not p['spec'].get('securityContext',{}).get('runAsUser')]; print(len(bad))")
[ "${ROOT_PODS:-0}" -eq "0" ] && pass "" || fail "$ROOT_PODS Pods without runAsNonRoot"
# 2.3 Default ServiceAccount automount disabled
echo -n "2.3 Default SA automount off: "
AUTOMOUNT=$(kubectl get sa default -n production -o jsonpath='{.automountServiceAccountToken}' 2>/dev/null)
[ "$AUTOMOUNT" = "false" ] && pass "" || fail "automount=$AUTOMOUNT"
# 2.4 Resource limits set on all containers
echo -n "2.4 Resource limits set: "
NO_LIMITS=$(kubectl get pods -l app=ipl-api -n production -o json | python3 -c "import sys,json; pods=json.load(sys.stdin)['items'];
bad=[c['name'] for p in pods for c in p['spec']['containers']
if not c.get('resources',{}).get('limits')]; print(len(bad))")
[ "${NO_LIMITS:-0}" -eq "0" ] && pass "" || fail "$NO_LIMITS containers without limits"
# 2.5 IRSA annotation on ServiceAccount
echo -n "2.5 IRSA annotation present: "
IRSA=$(kubectl get sa ipl-api-sa -n production -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}' 2>/dev/null)
[ -n "$IRSA" ] && pass "" || fail "IRSA annotation missing"
echo ""
echo "Supply Chain + Workload Security score: $SCORE/$MAX" Step 2 — Network, Storage and Observability Audit
# Dimensions 3, 4, and 5: Network + Storage + Observability
echo "=== DIMENSION 3: Network Security (10 points) ==="
# 3.1 Default-deny NetworkPolicy present
echo -n "3.1 Default-deny NetworkPolicy: "
kubectl get networkpolicy default-deny-all -n production > /dev/null 2>&1 && pass "" || fail ""
# 3.2 DNS egress allow-rule present
echo -n "3.2 DNS egress allow-rule: "
kubectl get networkpolicy allow-dns-egress -n production > /dev/null 2>&1 && pass "" || fail ""
# 3.3 Database port not published (no NodePort/LoadBalancer for postgres)
echo -n "3.3 Database port not exposed: "
EXPOSED_PG=$(kubectl get svc -n production -l app=ipl-postgres -o jsonpath='{.items[*].spec.type}' | grep -c "NodePort\|LoadBalancer" || true)
[ "${EXPOSED_PG:-0}" -eq "0" ] && pass "" || fail "postgres service exposed externally"
# 3.4 Ingress TLS configured
echo -n "3.4 Ingress has TLS: "
TLS_HOST=$(kubectl get ingress ipl-platform-ingress -n production -o jsonpath='{.spec.tls[0].hosts[0]}' 2>/dev/null)
[ -n "$TLS_HOST" ] && pass "host=$TLS_HOST" || fail "no TLS on Ingress"
# 3.5 cert-manager certificate Ready
echo -n "3.5 cert-manager cert Ready: "
CERT_READY=$(kubectl get certificate -n production -o jsonpath='{.items[0].status.conditions[0].status}' 2>/dev/null)
[ "$CERT_READY" = "True" ] && pass "" || fail "cert status=$CERT_READY"
echo ""
echo "=== DIMENSION 4: Storage and Data Durability (10 points) ==="
# 4.1 StatefulSet PVCs using Retain policy
echo -n "4.1 PVCs use Retain policy: "
DELETE_PVCS=$(kubectl get pv -o json | python3 -c "
import sys,json; pvs=json.load(sys.stdin)['items'];
bad=[pv['metadata']['name'] for pv in pvs
if pv['spec'].get('claimRef',{}).get('namespace')=='production'
and pv['spec']['persistentVolumeReclaimPolicy']=='Delete']; print(len(bad))")
[ "${DELETE_PVCS:-0}" -eq "0" ] && pass "" || fail "$DELETE_PVCS PVCs with Delete policy"
# 4.2 gp3-encrypted StorageClass exists
echo -n "4.2 Encrypted StorageClass: "
kubectl get storageclass gp3-encrypted > /dev/null 2>&1 && pass "" || fail "gp3-encrypted StorageClass missing"
# 4.3 Velero backup schedule active
echo -n "4.3 Velero backup schedule: "
velero schedule get 2>/dev/null | grep -q "ENABLED" && pass "" || fail "no active Velero schedule"
# 4.4 VolumeSnapshot CronJob present
echo -n "4.4 Snapshot CronJob: "
kubectl get cronjob -n production | grep -q "snapshot" && pass "" || fail "no snapshot CronJob"
# 4.5 StatefulSet using volumeClaimTemplate (not ephemeral storage)
echo -n "4.5 StatefulSet has volumeClaimTemplate: "
VCT=$(kubectl get statefulset ipl-postgres -n production -o jsonpath='{.spec.volumeClaimTemplates[0].metadata.name}' 2>/dev/null)
[ -n "$VCT" ] && pass "template=$VCT" || fail "no volumeClaimTemplate"
echo ""
echo "=== DIMENSION 5: Observability and Alerting (10 points) ==="
# 5.1 ServiceMonitor scraping the API
echo -n "5.1 ServiceMonitor configured: "
kubectl get servicemonitor ipl-api-metrics -n monitoring > /dev/null 2>&1 && pass "" || fail ""
# 5.2 PrometheusRule with SLO alerts
echo -n "5.2 PrometheusRule SLO alerts: "
kubectl get prometheusrule ipl-api-alerts -n monitoring > /dev/null 2>&1 && pass "" || fail ""
# 5.3 Prometheus successfully scraping API metrics
echo -n "5.3 Prometheus scraping API: "
kubectl port-forward -n monitoring svc/kube-prometheus-stack-prometheus 9090:9090 &
PF_PID=$!; sleep 3
METRIC=$(curl -s "http://localhost:9090/api/v1/query?query=http_requests_total{app='ipl-api'}" | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d['data']['result']))")
kill $PF_PID 2>/dev/null
[ "${METRIC:-0}" -gt "0" ] && pass "($METRIC series)" || fail ""
# 5.4 Grafana dashboard for IPL API
echo -n "5.4 Grafana IPL dashboard: "
kubectl get configmap -n monitoring -l grafana_dashboard=1 | grep -q "ipl-api" && pass "" || fail ""
# 5.5 Jaeger receiving traces
echo -n "5.5 Traces in Jaeger: "
kubectl port-forward -n monitoring svc/jaeger-query 16686:16686 &
PF_PID=$!; sleep 3
TRACE_COUNT=$(curl -s "http://localhost:16686/api/traces?service=ipl-api&limit=1&lookback=24h" | python3 -c "import sys,json; d=json.load(sys.stdin); print(len(d['data']))" 2>/dev/null)
kill $PF_PID 2>/dev/null
[ "${TRACE_COUNT:-0}" -gt "0" ] && pass "" || fail "" Step 3 — Deployment Automation Audit and Final Score
# Dimension 6: Deployment Automation + Final Score
echo ""
echo "=== DIMENSION 6: Deployment Automation (10 points) ==="
# 6.1 Argo CD Application exists and is Synced
echo -n "6.1 Argo CD Application Synced: "
SYNC=$(argocd app get ipl-platform -o json 2>/dev/null | python3 -c "import sys,json; d=json.load(sys.stdin); print(d['status']['sync']['status'])" 2>/dev/null)
[ "$SYNC" = "Synced" ] && pass "" || fail "sync=$SYNC"
# 6.2 selfHeal enabled
echo -n "6.2 selfHeal enabled: "
SELFHEAL=$(kubectl get application ipl-platform -n argocd -o jsonpath='{.spec.syncPolicy.automated.selfHeal}' 2>/dev/null)
[ "$SELFHEAL" = "true" ] && pass "" || fail "selfHeal=$SELFHEAL"
# 6.3 No direct kubectl apply in deployment history (GitOps-only changes)
echo -n "6.3 Deployment via Git commits: "
# Check that last 5 syncs all have a Git revision (not manual apply)
REVISIONS=$(argocd app history ipl-platform 2>/dev/null | grep -c "sha-" 2>/dev/null || echo "0")
[ "${REVISIONS:-0}" -gt "0" ] && pass "($REVISIONS Git-tagged deployments)" || fail ""
# 6.4 Sealed Secrets or External Secrets for credentials
echo -n "6.4 No plaintext Secrets in Git: "
# Check manifests repo for Secret objects with plaintext data
if [ -d "../ipl-platform-k8s" ]; then
PLAINTEXT_SECRETS=$(grep -r "kind: Secret" ../ipl-platform-k8s/k8s/ 2>/dev/null | grep -v SealedSecret | wc -l | tr -d ' ')
[ "${PLAINTEXT_SECRETS:-0}" -eq "0" ] && pass "" || fail "$PLAINTEXT_SECRETS plaintext Secrets in Git"
else
echo " SKIP (manifests repo not accessible locally)"
fi
# 6.5 CI pipeline includes build+scan+sign+push sequence
echo -n "6.5 CI pipeline: build+scan+sign: "
# Verify GitHub Actions workflow exists with all steps
if [ -f ".github/workflows/build-scan-push.yml" ]; then
grep -q "trivy" .github/workflows/build-scan-push.yml && grep -q "cosign" .github/workflows/build-scan-push.yml && grep -q "docker.*build" .github/workflows/build-scan-push.yml && pass "" || fail "CI missing steps"
else
fail "CI workflow not found"
fi
echo ""
echo "======================================"
echo "PRODUCTION READINESS AUDIT COMPLETE"
echo "======================================"
echo ""
echo "Final Score: $SCORE / 60"
echo ""
if [ "$SCORE" -ge "50" ]; then
echo "RESULT: PRODUCTION READY ✓"
echo "The IPL analytics platform meets production standards across all"
echo "six dimensions: supply chain, security, networking, storage,"
echo "observability, and deployment automation."
elif [ "$SCORE" -ge "40" ]; then
echo "RESULT: READY WITH MINOR GAPS"
echo "Address the failed items before production deployment."
else
echo "RESULT: NOT YET PRODUCTION READY"
echo "Critical gaps require remediation before production deployment."
fi Course Completion: Architecture Principles Map
The complete IPL analytics platform built across Course 3 demonstrates that every production Kubernetes system is an expression of a coherent set of container and orchestration principles working at different layers of abstraction simultaneously. The M1 OverlayFS understanding explains why `readOnlyRootFilesystem: true` is not a Kubernetes security setting but a filesystem property; the M1 OCI spec knowledge explains why every container runs identically from development to production; the M2 Dockerfile layer cache principles explain why the CI pipeline's dependency installation step completes in seconds rather than minutes; the M3 reconciliation model explains why deleting a Pod is never a destructive operation; and the M4 StorageClass WaitForFirstConsumer explains why the StatefulSet's data is always accessible after rescheduling. Each of these connections is the point at which a course lesson becomes a production debugging skill — the ability to diagnose a problem by understanding the mechanism that caused it.
The security stack built in M5 — RBAC, PSA, NetworkPolicies, Kyverno, Vault — implements a layered defence where each control is independently enforceable and independently bypassable only with explicit, audited escalation. A compromised container that escapes its namespace boundary through a container runtime vulnerability lands in a network that has no east-west access to databases, has no API server credentials beyond what its ServiceAccount explicitly grants, and cannot install tools in the container filesystem because it is read-only. No single control provides complete protection; the combination ensures that a successful exploit at one layer stops at the next. The observability and GitOps layers built in M6 ensure that any deviation from the designed state — whether from a misconfiguration, a security incident, or an infrastructure failure — is immediately visible in metrics, traceable through distributed traces, auditable through Argo CD's sync history, and recoverable through Velero backup restores.
Course Completion: You have built, secured, instrumented, and validated a production-grade containerised platform covering the complete container and Kubernetes lifecycle: from Linux namespaces and OCI specifications through container images, registries, and supply chain security; through Kubernetes workloads, services, storage, and autoscaling; through RBAC, Pod Security Standards, NetworkPolicies, and policy-as-code governance; through Prometheus observability, distributed tracing, GitOps deployment, and chaos engineering resilience validation. The IPL analytics platform running at the end of this course is a reference architecture that implements every principle introduced across 40 lessons in a coherent, auditable, production-deployable system. Congratulations on completing Course 3.
- The production readiness audit scores six dimensions — supply chain, workload security, network security, storage durability, observability, and deployment automation — providing a structured gap analysis before any production deployment.
- Every platform component traces to a course principle: readOnlyRootFilesystem to M1 OverlayFS, WaitForFirstConsumer to M4 AZ-aware storage, condition:service_healthy init container to M2 Compose healthcheck, Retain reclaim policy to M4 data durability.
- The layered security model — RBAC, PSA, NetworkPolicies, Kyverno — ensures that a compromise at any single layer is contained by the controls at the next layer, with no single bypass producing unlimited blast radius.
- Observability (metrics, traces, logs) and GitOps (Argo CD, Velero) together make the platform diagnosable, auditable, and recoverable — the three properties that distinguish a cluster that works from a cluster that is operationally trustworthy.
- Chaos engineering with LitmusChaos provides empirical evidence of resilience properties rather than theoretical declarations — a pod-delete experiment that passes is evidence that availability is maintained during failure, not just assumed.
- The complete Course 3 platform is a composition of Kubernetes primitives — not a set of independently configured tools but an integrated architecture where each layer's design decisions are informed by and consistent with the layers above and below it.