Practice — StatefulSet PostgreSQL with HPA on the API tier

What You'll Build

In this exercise you will upgrade the IPL analytics platform from M3 Exercise to use production-grade storage and autoscaling. You will replace the simple PostgreSQL Deployment with a StatefulSet backed by a `Retain`-policy StorageClass and a `volumeClaimTemplate`, take a VolumeSnapshot of the primary's data before migrating, and configure an HPA on the API tier with CPU and custom RPS metrics. You will verify that the StatefulSet's stable identity works correctly by confirming that PostgreSQL data survives a Pod deletion and rescheduling, that the HPA scales the API Deployment in response to a load test, and that the HPA's stabilisation window prevents immediate scale-down when the load stops. The complete platform after this exercise is production-ready for the Ingress and TLS lab that follows.

Analogy🏏Cricket

🏏 Think of it like cricket: This lab's staging-then-production TLS pipeline is the ICC's pre-tour practice match protocol. Before the Test series proper, the touring team plays a two-day warm-up match against a local state side — not under ICC playing conditions, not with the official match balls, and not counted in official records. The warm-up match validates that the team's batting and bowling combinations work on local pitch conditions before committing to the conditions for the official five-day Test. The Let's Encrypt staging environment is that warm-up match: it issues real certificates signed by a staging CA (not trusted by browsers, like an unofficial match result), validates the complete ACME challenge flow, and confirms that DNS, network, and IAM configurations are correct — all without consuming the production rate limit. Switching to the production issuer is committing to the official Test: the certificate is now signed by Let's Encrypt's trusted CA (officially recognised), but any misconfiguration wastes an official certificate issuance. The certificate rotation simulation is the team testing their emergency substitution protocol — specifically waiting until the 'player' (certificate) is within the renewal window, confirming the automatic replacement process fires correctly, and verifying the new 'player' is ready to take the field. This reveals why the three-phase structure of the lab matters: each phase validates a distinct property of the TLS chain, and proceeding through them in order reduces the risk that a configuration problem discovered in production was one that staging testing would have caught.

Prerequisites

The IPL analytics platform from M3 Exercise running with a simple PostgreSQL Deployment — the existing data will be migrated to the StatefulSet using a VolumeSnapshot.
The EBS CSI driver installed and a `gp3-encrypted` StorageClass configured with `reclaimPolicy: Retain` and `volumeBindingMode: WaitForFirstConsumer` from Lesson 25.
The Kubernetes Metrics Server installed and verified with `kubectl top pods -n production` returning CPU and memory values before beginning — HPA requires metrics-server to function.
The VolumeSnapshot CRDs and the external-snapshotter controller installed — verify with `kubectl get crd | grep snapshot` showing `volumesnapshots.snapshot.storage.k8s.io`.
A load testing tool available: `kubectl run load-test --image=williamyeh/wrk --rm -it` or `hey` installed locally for the HPA scaling verification step.

Setup & Project Structure

Before replacing the PostgreSQL Deployment with a StatefulSet, snapshot the existing data volume to enable rollback if the migration encounters problems. The snapshot is the insurance policy — a production migration that changes the storage layer without a backup is an unacceptable risk regardless of how simple the migration appears. After snapshotting, scale the existing Deployment to zero replicas (preventing writes to the data), then create the StatefulSet which will mount the same data via a PVC seeded from the snapshot.

Analogy🏏Cricket

🏏 Think of it like cricket: setting up a tournament venue is not one job but a strict sequence of jobs, and smart boards hire a professional event crew instead of doing each task by hand. Just as the event crew handles seating, accreditation desks, and broadcast rigging as one coordinated package, Helm installs the Nginx IngressController and cert-manager with all their CRDs, ServiceAccounts, and RBAC in one release instead of dozens of hand-applied manifests. And order matters: just as the DRS cameras and replay screens must be rigged and tested before the third umpire takes his seat — an official with no working replay feed is useless and the review system collapses — cert-manager's CRDs must exist before the controller starts, or it crashes on startup before its CRD admission webhooks ever come online. The payoff: a correctly sequenced, fully provisioned venue — controller infrastructure that comes up cleanly the first time.

bash

# Migration setup: snapshot existing data before replacing Deployment

# ── Step 1: Identify the existing PostgreSQL PVC ───────────────────────────
kubectl get pvc -n production -l app=ipl-postgres
# NAME                STATUS  VOLUME          CAPACITY
# ipl-postgres-data   Bound   pvc-abc123...   5Gi

# ── Step 2: Create a VolumeSnapshot before migration ──────────────────────
kubectl apply -f - << 'EOF'
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: postgres-pre-migration
  namespace: production
spec:
  volumeSnapshotClassName: ebs-snapshots
  source:
    persistentVolumeClaimName: ipl-postgres-data
EOF

kubectl wait volumesnapshot/postgres-pre-migration   -n production   --for=jsonpath='{.status.readyToUse}'=true   --timeout=120s
echo "Snapshot ready ✓"

# ── Step 3: Scale existing Deployment to zero (stop writes) ────────────────
kubectl scale deployment ipl-postgres --replicas=0 -n production
kubectl wait deployment/ipl-postgres   --for=jsonpath='{.status.availableReplicas}'=0   -n production --timeout=60s
echo "Deployment scaled to zero — no new writes to data ✓"

# ── Step 4: Create the StorageClass for StatefulSet ───────────────────────
kubectl apply -f - << 'EOF'
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3-encrypted
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  encrypted: "true"
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
EOF

echo "Migration prerequisites complete. Ready to create StatefulSet."
echo "Rollback available via: postgres-pre-migration VolumeSnapshot"

Step 1 — Foundation

Create the PostgreSQL StatefulSet with a headless Service and a `volumeClaimTemplate` seeded from the pre-migration snapshot. The first StatefulSet PVC is pre-populated from the snapshot, so when `ipl-postgres-0` starts, it finds the existing data directory with all previous data intact — exactly as PostgreSQL left it when the Deployment was scaled to zero. Verify stable identity by confirming the Pod name and hostname match, then confirm the pre-existing data is visible inside the new StatefulSet Pod.

Analogy🏏Cricket

🏏 Think of it like cricket: before the first official fixture, a serious team plays a full-dress practice match — real pitch, real umpires, real match conditions — knowing the result won't appear in any league table. Just as that practice match earns no points but proves the batting order, bowling plans, and fielding drills actually work under match pressure, the Let's Encrypt staging certificate won't be trusted by any browser but proves that the ACME challenge completes, DNS resolves correctly, and the IngressController routes traffic as designed. Just as a coach would never debut an untested game plan in a televised final, you never point at the production issuer first — a failed live attempt costs far more than a failed rehearsal. The payoff: every moving part of the certificate pipeline validated cheaply, so the later switch to production is a formality rather than a gamble.

bash

# StatefulSet PostgreSQL with snapshot-seeded first PVC

kubectl apply -f - << 'EOF'
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: ipl-postgres
  namespace: production
spec:
  serviceName: ipl-postgres
  replicas: 1
  selector:
    matchLabels: {app: ipl-postgres}
  template:
    metadata:
      labels: {app: ipl-postgres}
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: postgres
          image: postgres:16-alpine
          env:
            - name: POSTGRES_DB
              value: cricket_stats
            - name: POSTGRES_USER
              value: rohit_admin
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: ipl-postgres-secret
                  key: password
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata  # avoid lost+found directory issue
          resources:
            requests: {cpu: 200m, memory: 256Mi}
            limits: {cpu: "1", memory: 1Gi}
          readinessProbe:
            exec:
              command: ["pg_isready", "-U", "rohit_admin", "-d", "cricket_stats"]
            initialDelaySeconds: 10
            periodSeconds: 5
          volumeMounts:
            - name: postgres-data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata:
        name: postgres-data
      spec:
        storageClassName: gp3-encrypted
        accessModes: [ReadWriteOnce]
        resources:
          requests: {storage: 10Gi}
        # Seed from migration snapshot — data migrated automatically
        dataSource:
          name: postgres-pre-migration
          kind: VolumeSnapshot
          apiGroup: snapshot.storage.k8s.io
---
apiVersion: v1
kind: Service
metadata:
  name: ipl-postgres
  namespace: production
spec:
  clusterIP: None
  selector: {app: ipl-postgres}
  ports: [{port: 5432}]
EOF

# Wait for PostgreSQL to be ready
kubectl wait pod/ipl-postgres-0   -n production --for=condition=Ready --timeout=120s

# Verify stable identity
echo -n "Pod hostname: "
kubectl exec ipl-postgres-0 -n production -- hostname
# Expected: ipl-postgres-0

# Verify migrated data is intact
echo -n "Existing data: "
kubectl exec ipl-postgres-0 -n production --   psql -U rohit_admin -d cricket_stats -t   -c "SELECT COUNT(*) FROM ipl_test;"
# Expected: non-zero row count from M3 Exercise data  ✓

# Delete existing Deployment (no longer needed)
kubectl delete deployment ipl-postgres -n production 2>/dev/null || true
echo "Migration complete ✓"

Step 2 — Core Logic

Configure the HPA on the API Deployment with CPU utilisation and custom RPS targets, then run a load test to trigger scaling and observe the HPA's behaviour. Document the actual replica counts at each step of the load test — start, peak, post-load stabilisation, and final scale-down — to verify that the scaling formula, the stabilisation window, and the scale-down rate limit all work as designed. The load test output, combined with the HPA event history, is the evidence that the autoscaling configuration is correctly calibrated for this workload.

Analogy🏏Cricket

🏏 Think of it like cricket: once the practice match proves the plans work, the team steps into the official fixture — and everything must now count for real. Just as the practice-game scorecard is torn up so the official scorers start a fresh book, the staging certificate Secret is deleted so cert-manager issues a fresh production certificate rather than serving the old untrusted one. Just as the umpires formally verify the match ball and playing conditions before play begins, you verify the production certificate against the system certificate store — curl without -k must succeed. Then comes rehearsing the handover: just as a club renews a key player's contract well before it expires so there is never a day without him under contract, patching renewBefore to 89 days forces cert-manager to renew the 90-day certificate almost immediately, proving rotation happens automatically long before expiry. The payoff: trusted TLS in production plus demonstrated automatic renewal — no midnight expiry emergencies.

bash

# HPA configuration and load test verification

kubectl apply -f - << 'EOF'
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ipl-api-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ipl-api
  minReplicas: 2
  maxReplicas: 8
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: {type: Utilization, averageUtilization: 60}
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 120   # reduced for lab visibility (normally 300s)
      policies:
        - {type: Pods, value: 1, periodSeconds: 30}
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - {type: Pods, value: 2, periodSeconds: 30}
EOF

# ── Run load test and observe HPA in real time ─────────────────────────────
# Terminal 1: watch HPA
kubectl get hpa ipl-api-hpa -n production -w &
HPA_WATCH=$!

# Terminal 2: load test — 10 concurrent workers for 60 seconds
kubectl run load-test   --image=williamyeh/wrk   --rm   --restart=Never   -n production   -- -t2 -c50 -d60s http://ipl-api.production.svc.cluster.local:8000/batters

sleep 30   # allow HPA to detect and respond to the load

echo "=== HPA status during load ==="
kubectl get hpa ipl-api-hpa -n production
kubectl get pods -l app=ipl-api -n production

echo ""
echo "=== Waiting for load to stop and stabilisation window to expire ==="
sleep 150   # wait for 120s stabilisation window + buffer

echo "=== HPA status after stabilisation ==="
kubectl get hpa ipl-api-hpa -n production
# Expected: replicas back to minReplicas=2 after 120s stabilisation

kill $HPA_WATCH 2>/dev/null

# Inspect HPA event log for scaling decisions
kubectl describe hpa ipl-api-hpa -n production | grep -A20 "Events:"
# Events:
#   Normal  SuccessfulRescale  New size: 4; reason: cpu above target
#   Normal  SuccessfulRescale  New size: 2; reason: All metrics below target

Step 3 — Integration & Enhancement

Verify StatefulSet stable identity by deleting `ipl-postgres-0` and confirming that the replacement Pod rejoins with the same name and the same data — the two properties that distinguish a StatefulSet from a Deployment. Add a VolumeSnapshot CronJob that takes nightly snapshots of the PostgreSQL PVC for disaster recovery, completing the production storage posture for this platform.

Analogy🏏Cricket

🏏 Think of it like cricket: on the eve of a tournament the venue runs a full walk-through — a spectator's journey from the car park, through the ticket gate, to the correct stand, with stewards redirecting anyone who wanders toward the wrong entrance. Just as that walk-through exercises every link in the chain, the end-to-end check verifies DNS resolution, the TLS handshake with the production certificate, routing to the correct backend service, and the HTTPS redirect that steers plain-HTTP stragglers to the secure entrance. Just as gate staff read the stand printed on each ticket and route spectators accordingly, host-based routing reads the hostname and sends pgAdmin traffic to its own backend while API traffic goes to the API. And just as an all-venue tournament pass admits its holder to every ground without a separate ticket per stadium, the wildcard certificate covers both subdomains with a single credential. The payoff: one entry point, correctly securing and routing every kind of visitor.

bash

# StatefulSet identity persistence + nightly snapshot CronJob

# ── Test stable identity: delete and verify recreation ─────────────────────
# Insert marker data to verify recovery
kubectl exec ipl-postgres-0 -n production --   psql -U rohit_admin -d cricket_stats -c   "INSERT INTO ipl_test(message) VALUES ('Pre-delete marker: $(date)');"

# Delete the Pod (StatefulSet will recreate it with same name)
kubectl delete pod ipl-postgres-0 -n production

# Watch recreation — Pod should come back as ipl-postgres-0, not a random name
kubectl get pods -l app=ipl-postgres -n production -w
# ipl-postgres-0   Running → Terminating → Pending → ContainerCreating → Running

# Verify data persisted through deletion/recreation
kubectl wait pod/ipl-postgres-0 -n production --for=condition=Ready --timeout=120s
kubectl exec ipl-postgres-0 -n production --   psql -U rohit_admin -d cricket_stats -t   -c "SELECT message FROM ipl_test ORDER BY inserted_at DESC LIMIT 3;"
# Shows "Pre-delete marker" entry — data survived Pod recreation ✓

# ── CronJob: nightly VolumeSnapshot for disaster recovery ─────────────────
kubectl apply -f - << 'EOF'
apiVersion: batch/v1
kind: CronJob
metadata:
  name: ipl-postgres-snapshot
  namespace: production
spec:
  schedule: "0 1 * * *"    # 01:00 UTC = 06:30 IST
  timeZone: "UTC"
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 7
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 1
      activeDeadlineSeconds: 600
      template:
        spec:
          restartPolicy: Never
          serviceAccountName: snapshot-manager    # needs VolumeSnapshot create permission
          containers:
            - name: snapshot
              image: bitnami/kubectl:latest
              command:
                - sh
                - -c
                - |
                  DATE=$(date +%Y%m%d)
                  kubectl apply -f - << SNAP
                  apiVersion: snapshot.storage.k8s.io/v1
                  kind: VolumeSnapshot
                  metadata:
                    name: postgres-nightly-$DATE
                    namespace: production
                  spec:
                    volumeSnapshotClassName: ebs-snapshots
                    source:
                      persistentVolumeClaimName: postgres-data-ipl-postgres-0
                  SNAP
                  echo "Snapshot postgres-nightly-$DATE created"
EOF

echo "Nightly snapshot CronJob created ✓"
kubectl get cronjob ipl-postgres-snapshot -n production

Step 4 — Testing & Verification

Run the final verification checklist confirming all M4 Exercise properties: StatefulSet stable identity, PVC data persistence, HPA scaling event history, snapshot existence, and the API serving correct responses after the complete storage migration and autoscaling configuration. The checklist confirms the platform is ready for the Ingress and TLS lab.

Analogy🏏Cricket

🏏 Think of it like cricket: at the midpoint of a season, a good head coach doesn't just check the points table — he traces how every department produced it: the academy that developed the players, the selectors who picked the squad, and the matchday operations that got the team onto the field. Just as that review makes the whole club visible as one system rather than isolated departments, the M1–M4 architecture summary traces each component's role — the container image layer where the application is built and packaged, the Kubernetes workload layer where Deployments, StatefulSets, and autoscaling run it, and the external access layer where Ingress and TLS expose it to the world. Just as ticking the final checklist confirms match-readiness, completing the lab checklist confirms every layer actually works together. The payoff: you can explain the entire platform end to end — the mark of real understanding, and the foundation M5's security work builds on.

bash

# Final verification checklist

echo "=== M4 Exercise — StatefulSet & HPA Verification ==="

# 1. StatefulSet Pod has stable name
echo -n "1. StatefulSet stable name:           "
POD_NAME=$(kubectl get pod -l app=ipl-postgres -n production   -o jsonpath='{.items[0].metadata.name}')
[ "$POD_NAME" = "ipl-postgres-0" ] && echo "PASS ✓ ($POD_NAME)" || echo "FAIL ✗"

# 2. PVC bound and using Retain policy
echo -n "2. PVC bound with Retain policy:      "
RECLAIM=$(kubectl get pv   $(kubectl get pvc postgres-data-ipl-postgres-0 -n production     -o jsonpath='{.spec.volumeName}')   -o jsonpath='{.spec.persistentVolumeReclaimPolicy}')
[ "$RECLAIM" = "Retain" ] && echo "PASS ✓" || echo "FAIL ✗ (got: $RECLAIM)"

# 3. Data persists through Pod deletion
echo -n "3. Data survived Pod deletion:        "
ROWS=$(kubectl exec ipl-postgres-0 -n production --   psql -U rohit_admin -d cricket_stats -t   -c "SELECT COUNT(*) FROM ipl_test;" 2>/dev/null | tr -d ' ')
[ "$ROWS" -gt "0" ] && echo "PASS ✓ ($ROWS rows)" || echo "FAIL ✗"

# 4. HPA exists and is in a valid state
echo -n "4. HPA configured correctly:          "
HPA_MIN=$(kubectl get hpa ipl-api-hpa -n production   -o jsonpath='{.spec.minReplicas}')
[ "$HPA_MIN" = "2" ] && echo "PASS ✓ (minReplicas=$HPA_MIN)" || echo "FAIL ✗"

# 5. HPA scaled up during load test (check event history)
echo -n "5. HPA scaled up during load test:    "
kubectl describe hpa ipl-api-hpa -n production | grep -q "SuccessfulRescale"   && echo "PASS ✓" || echo "FAIL ✗ — run load test first"

# 6. VolumeSnapshot exists
echo -n "6. Migration snapshot present:        "
kubectl get volumesnapshot postgres-pre-migration -n production > /dev/null 2>&1   && echo "PASS ✓" || echo "FAIL ✗"

# 7. API responding correctly
echo -n "7. API serves correct responses:      "
kubectl exec -n production   $(kubectl get pod -l app=ipl-api -n production -o name | head -1)   -- python3 -c "import urllib.request; print(urllib.request.urlopen('http://localhost:8000/health').read())"   | grep -q "healthy" && echo "PASS ✓" || echo "FAIL ✗"

echo "=== Exercise complete ==="

Warning: The `PGDATA: /var/lib/postgresql/data/pgdata` environment variable is required when using EBS volumes with the official PostgreSQL image because EBS volumes mount a root directory that may contain a `lost+found` directory, which PostgreSQL refuses to use as its data directory. Setting PGDATA to a subdirectory avoids this constraint. Without this variable, the StatefulSet Pod will repeatedly fail with 'initdb: error: directory /var/lib/postgresql/data is not empty — lost+found exists' and enter CrashLoopBackOff. This is not a Kubernetes issue — it is the PostgreSQL initialisation process correctly refusing to use a non-empty directory — but the fix is a one-line environment variable, not a volume configuration change.

Extension Challenge: Add a PostgreSQL read replica as a second StatefulSet replica (`replicas: 2`) and configure streaming replication from `ipl-postgres-0` (primary) to `ipl-postgres-1` (standby) using the `POSTGRES_REPLICATION_USER` and `POSTGRES_REPLICATION_PASSWORD` environment variables. Update the API Deployment to route read-only queries to `ipl-postgres-1.ipl-postgres.production.svc.cluster.local` and write queries to `ipl-postgres-0.ipl-postgres.production.svc.cluster.local` using SQLAlchemy's read/write splitting. This exercises the headless Service stable DNS, the StatefulSet ordered creation (primary must be Ready before replica starts), and the Downward API ordinal detection for primary vs replica role assignment.

Always snapshot the existing data volume before migrating from a Deployment to a StatefulSet — the snapshot is the rollback mechanism if the migration encounters unexpected issues.
The `dataSource: volumeSnapshot` field in a PVC spec pre-populates the new PVC from a snapshot, enabling zero-data-loss storage migrations without manual pg_dump/pg_restore.
StatefulSet stable identity is verified by confirming that a deleted Pod reappears with the same name (ipl-postgres-0) and rebinds to the same PVC, finding its previous data intact.
HPA scaling events are preserved in kubectl describe hpa output — review them after a load test to confirm the scaling formula, stabilisation window, and rate limit policies work as designed.
Set `PGDATA` to a subdirectory (e.g. /var/lib/postgresql/data/pgdata) when using EBS block devices to avoid PostgreSQL refusing to initialise in a directory containing the EBS-created lost+found entry.
The nightly VolumeSnapshot CronJob completes the production storage posture: StatefulSet stable identity for data continuity, Retain reclaim policy for PVC protection, and automated snapshots for point-in-time recovery.