Capstone — Deploy CricketPulse with Match-Day Autoscaling

This capstone project deploys CricketPulse — a live cricket scores and statistics platform — as a complete production-grade Kubernetes application that handles the extreme traffic variability of live cricket: quiet baseline traffic between matches and 10-50× surge traffic during IPL finals. You will build a fully operational CricketPulse cluster: a PostgreSQL StatefulSet for persistent match data, a Redis Deployment for live score caching, a CricketPulse API Deployment exposed via Ingress, Prometheus and Grafana for observability, HPA for automatic match-day scaling, PDBs for maintenance safety, and everything packaged in a Helm chart with environment-specific values for dev and production. The result is a portfolio-quality Kubernetes deployment that demonstrates every concept from this course working together in a realistic, demanding scenario.

Learning Objectives

• Compose a complete multi-tier application on Kubernetes: StatefulSet database, Deployment-based microservices, ConfigMaps and Secrets for configuration, all interconnected via Services and DNS.
• Package the entire application as a Helm chart with parameterised templates and separate production and development values files.
• Configure HPA with custom CPU targets and conservative scale-down behaviour to handle IPL match-day traffic spikes without oscillation.
• Implement proper observability: Prometheus metrics endpoint, structured JSON logging, and kube-state-metrics-compatible Service annotations.
• Apply RBAC with dedicated ServiceAccounts, minimal permissions, and namespace isolation between environments.
• Validate the complete deployment by simulating a match-day traffic spike and observing HPA scale-up, then verifying production safeguards (PDB) prevent simultaneous Pod eviction during maintenance.

Technical Requirements

• Three Kubernetes namespaces: `cricketpulse-dev` (no protection, minimal resources), `cricketpulse-prod` (ResourceQuota, LimitRange), and `monitoring` (Prometheus + Grafana via Helm chart).
• PostgreSQL StatefulSet (1 replica in dev, 3 in prod) with a headless Service, dedicated PVC per Pod via `volumeClaimTemplates`, and database credentials from a Kubernetes Secret.
• Redis Deployment (1 replica in dev, 2 in prod) for live score caching, exposed as a ClusterIP Service.
• CricketPulse API Deployment (2 replicas in dev, 3+ in prod) with CPU/memory resource requests, readiness and liveness probes, Prometheus metrics port, and dedicated ServiceAccount with minimal RBAC.
• HPA on the CricketPulse API: `minReplicas: 2, maxReplicas: 20, targetCPU: 60%` with conservative scale-down behavior (5 minute stabilisation window).
• Ingress with path-based routing (`/api` → cricketpulse-api Service, `/` → cricketpulse-frontend Service) and TLS if a domain is available.
• Pod Disruption Budget on the API: `minAvailable: 2`.
• Helm chart packaging all resources with `values-dev.yaml` and `values-prod.yaml` override files.

Architecture & Design

The CricketPulse architecture is a three-tier application: data tier (PostgreSQL StatefulSet), cache tier (Redis Deployment), and application tier (CricketPulse API Deployment). Each tier communicates via Kubernetes ClusterIP Services and DNS service discovery — the API reaches PostgreSQL at `cricketpulse-db.cricketpulse-prod.svc.cluster.local:5432` and Redis at `cricketpulse-redis:6379`. External traffic enters through the Ingress controller, which routes to the API and frontend Services. The observability stack (Prometheus + Grafana) runs in a dedicated `monitoring` namespace and scrapes metrics from all three tiers. RBAC is configured with three ServiceAccounts: `cricketpulse-api` (read ConfigMaps, no other permissions), `cricketpulse-db` (no API permissions, `automountServiceAccountToken: false`), and `prometheus` (cluster-wide read access to discover and scrape metrics).

# CricketPulse Helm chart structure
cricketpulse/
  Chart.yaml
  values.yaml          # Default (dev) values
  values-prod.yaml     # Production overrides
  templates/
    _helpers.tpl
    namespace.yaml
    postgres/
      secret.yaml
      headless-service.yaml
      statefulset.yaml
    redis/
      deployment.yaml
      service.yaml
    api/
      serviceaccount.yaml
      role.yaml
      rolebinding.yaml
      configmap.yaml
      deployment.yaml
      service.yaml
      hpa.yaml
      pdb.yaml
    ingress.yaml
    resourcequota.yaml
    limitrange.yaml

# Values.yaml structure
namespace: cricketpulse-dev

postgres:
  replicas: 1
  storage: 5Gi
  storageClass: standard

redis:
  replicas: 1

api:
  image:
    repository: ghcr.io/srihayavadhana/cricketpulse
    tag: v1.4.2
  replicas: 2
  resources:
    requests: { cpu: 100m, memory: 256Mi }
    limits:   { cpu: 500m, memory: 512Mi }
  hpa:
    enabled: true
    minReplicas: 2
    maxReplicas: 10
    targetCPUUtilizationPercentage: 60
  pdb:
    enabled: true
    minAvailable: 2

ingress:
  enabled: true
  host: cricketpulse.local

Phase 1 — Core Implementation

Implement the data tier (PostgreSQL StatefulSet with headless Service and PVC template), cache tier (Redis Deployment with ClusterIP Service), and the CricketPulse API Deployment with ServiceAccount, RBAC, ConfigMap, and Services. Validate connectivity between all three tiers using kubectl exec.

# Phase 1: Deploy core services and verify connectivity

# 1. Install the cricketpulse Helm chart (dev environment)
helm install cricketpulse-dev ./cricketpulse \
  --namespace cricketpulse-dev \
  --create-namespace

# Wait for all components to be ready
kubectl wait deployment cricketpulse-api \
  --for=condition=Available --timeout=120s \
  -n cricketpulse-dev

kubectl rollout status statefulset/cricketpulse-db \
  -n cricketpulse-dev

# 2. Verify all components are running
kubectl get all,pvc,configmap,secret \
  -n cricketpulse-dev

# 3. Test inter-service connectivity
# API → PostgreSQL (via DNS)
API_POD=$(kubectl get pods -n cricketpulse-dev \
  -l app=cricketpulse-api \
  -o jsonpath='{.items[0].metadata.name}')

kubectl exec $API_POD -n cricketpulse-dev -- \
  nc -zv cricketpulse-db.cricketpulse-dev.svc.cluster.local 5432
# cricketpulse-db.cricketpulse-dev.svc.cluster.local (10.96.x.x:5432) open

# API → Redis (via DNS)
kubectl exec $API_POD -n cricketpulse-dev -- \
  nc -zv cricketpulse-redis 6379
# cricketpulse-redis (10.96.x.x:6379) open

# 4. Verify RBAC — confirm SA can only read ConfigMaps
kubectl auth can-i get configmaps \
  --namespace cricketpulse-dev \
  --as system:serviceaccount:cricketpulse-dev:cricketpulse-api
# yes

kubectl auth can-i delete pods \
  --namespace cricketpulse-dev \
  --as system:serviceaccount:cricketpulse-dev:cricketpulse-api
# no  ← Correctly restricted

Phase 2 — Feature Completion

Deploy to production with the production values (3 PostgreSQL replicas, 3 API replicas, production resource limits, production StorageClass), configure the observability stack (kube-prometheus-stack), and verify the HPA is operational by running a simulated match-day load test.

# Production values override file
cat > cricketpulse/values-prod.yaml << 'EOF'
namespace: cricketpulse-prod

postgres:
  replicas: 3
  storage: 50Gi
  storageClass: gp3-retain    # Production StorageClass with Retain policy

redis:
  replicas: 2

api:
  image:
    tag: v1.4.2
  replicas: 3
  resources:
    requests: { cpu: 200m, memory: 512Mi }
    limits:   { cpu: 1000m, memory: 1Gi }
  hpa:
    enabled: true
    minReplicas: 3
    maxReplicas: 30
    targetCPUUtilizationPercentage: 60
  pdb:
    enabled: true
    minAvailable: 3

ingress:
  enabled: true
  host: api.cricketpulse.com
EOF

# Deploy production
helm install cricketpulse-prod ./cricketpulse \
  --namespace cricketpulse-prod \
  --create-namespace \
  --values cricketpulse/values-prod.yaml

# Install monitoring stack
helm repo add prometheus-community \
  https://prometheus-community.github.io/helm-charts
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set grafana.adminPassword=cricketpulse-admin-2024

# Run load test to trigger HPA
kubectl port-forward svc/cricketpulse-api 8080:80 \
  -n cricketpulse-prod &
hey -z 120s -c 20 http://localhost:8080/v1/scores &

# Watch HPA scale-up in real time
watch kubectl get hpa,pods -n cricketpulse-prod

Phase 3 — Polish & Production Readiness

Validate the production safeguards by simulating a node drain and verifying the PDB correctly prevents simultaneous Pod eviction. Verify the rollback procedure by simulating a bad Helm upgrade. Run the final production readiness checklist confirming all components of the complete CricketPulse deployment are correctly configured and operational.

# Phase 3: Production safety validation

# 1. Test PDB restricts simultaneous eviction
# First: check how many disruptions are currently allowed
kubectl get pdb -n cricketpulse-prod
# cricketpulse-api-pdb: ALLOWED DISRUPTIONS = 1 (with 4 replicas, minAvailable: 3)

# Attempt to drain a worker node (should be restricted by PDB)
kubectl drain cricketpulse-cluster-worker2 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --dry-run=client   # DRY RUN first
# Observe: drain would be throttled by PDB to 1 Pod at a time

# 2. Test rollback procedure with a deliberately bad upgrade
helm upgrade cricketpulse-prod ./cricketpulse \
  --namespace cricketpulse-prod \
  --values cricketpulse/values-prod.yaml \
  --set api.image.tag=v99-DOES-NOT-EXIST  # Broken image

# Watch the deployment stall
kubectl get pods -n cricketpulse-prod -w &
timeout 60 kubectl rollout status deployment/cricketpulse-api \
  -n cricketpulse-prod || echo 'Rollout stalled!'

# Rollback to last good release
helm rollback cricketpulse-prod -n cricketpulse-prod
kubectl rollout status deployment/cricketpulse-api \
  -n cricketpulse-prod

# 3. Production readiness checklist
echo '=== CricketPulse Production Readiness ==='
kubectl get deployment,statefulset,hpa,pdb,ingress -n cricketpulse-prod
kubectl get pvc -n cricketpulse-prod
kubectl get endpoints -n cricketpulse-prod
kubectl top pods -n cricketpulse-prod
helm list --all-namespaces

Evaluation Rubric

• All three tiers (PostgreSQL StatefulSet, Redis Deployment, CricketPulse API Deployment) are running with correct Services and DNS names resolving between them — verified with `nc -zv` from the API Pod.
• Helm chart successfully installs to both dev namespace (default values) and prod namespace (values-prod.yaml overrides) with different resource configurations as specified.
• HPA scales the API Deployment above `minReplicas` during the load test and returns to `minReplicas` after the load test ends (within 5–10 minutes for scale-down).
• PDB correctly shows `ALLOWED DISRUPTIONS: 1` (not 0 or unlimited) with 4+ replicas and `minAvailable: 3`, and the PDB label selector matches the API Deployment's Pod labels exactly.
• RBAC verification passes: the `cricketpulse-api` ServiceAccount can `get` ConfigMaps but cannot `delete` Pods — confirmed with `kubectl auth can-i` for both verbs.
• Helm rollback (`helm rollback cricketpulse-prod`) successfully reverts to the previous chart revision and the deployment stabilises — confirmed with `kubectl rollout status`.
• PostgreSQL data persists through a Pod deletion — write a row to the database, delete the Pod, wait for restart, verify the row is still present via `kubectl exec psql`.