State Practice — Run a Stateful Database

What You'll Build

You will deploy a PostgreSQL database as a Kubernetes StatefulSet with a headless Service, per-Pod PVCs via `volumeClaimTemplates`, and environment configuration via Secrets and ConfigMaps. You will write data to the database, delete the Pod, verify the data persists when the Pod is recreated, and confirm that the stable DNS names work correctly. You will also deploy a simple CricketPulse API application connected to the database and verify the complete data flow from API to database and back.

Analogy🏏Cricket

🏏 Think of it like cricket: This exercise is a complete match simulation — not just batting practice. You'll face the full sequence: opening innings (initial deployment), the change of tactics mid-game (rolling update with canary check), an unexpected top-order collapse (bad deployment), and the team's recovery (rollback). Each scenario tests a different aspect of your Kubernetes deployment skills. Just as a batsman who's only practiced in nets needs match experience to handle the unexpected pressure of a live game, you need to experience deployment, update, failure, and recovery in sequence to build the operational instincts for real production management.

Prerequisites

A running kind cluster with storage support — kind uses the local-path-provisioner StorageClass by default, which supports RWO volumes on a single node.
kubectl configured and pointing to the kind cluster.
Lessons 13–15 completed — understanding of ConfigMaps, Secrets, PVCs, and StatefulSets.
Docker installed for image pull (postgres:16-alpine and hashicorp/http-echo images).
Basic familiarity with SQL or willingness to run provided `psql` commands.

Setup & Project Structure

Create the state exercise directory, verify the default StorageClass exists on the kind cluster (required for dynamic PVC provisioning), and create the database credentials Secret.

Analogy🏏Cricket

🏏 Think of it like cricket: Simulating different CricketPulse API versions with a ConfigMap and stock nginx is like rehearsing a batting-order change using the same players wearing different bib numbers, instead of recruiting brand-new specialists for every drill. Just as swapping a player's instructions on a laminated card lets the coach test how the side responds to a new plan without signing anyone, injecting different response text through a ConfigMap lets nginx serve 'v1' or 'v2' without building a custom Docker image. Just as a good academy sets out a clear practice plan — where each drill lives, which card belongs to which scenario — you lay out a tidy manifest directory before starting. The payoff: you can rehearse realistic multi-version rollout behaviour quickly and cheaply, learning the mechanics without the overhead of building and pushing real container images.

bash

mkdir -p cricketpulse-k8s/state
cd cricketpulse-k8s

# Verify kind has a default StorageClass (local-path)
kubectl get storageclass
# NAME                 PROVISIONER             RECLAIMPOLICY
# standard (default)   rancher.io/local-path   Delete

# Create the namespace
kubectl create namespace cricketpulse-db

# Create database credentials Secret
kubectl create secret generic postgres-secret \
  --from-literal=POSTGRES_PASSWORD=cricketpulse2024 \
  --from-literal=POSTGRES_DB=cricketpulse \
  --from-literal=POSTGRES_USER=cricketer \
  -n cricketpulse-db

# Create database configuration ConfigMap
kubectl create configmap postgres-config \
  --from-literal=MAX_CONNECTIONS=100 \
  --from-literal=SHARED_BUFFERS=256MB \
  -n cricketpulse-db

echo 'Namespace and credentials created'

Step 1 — Foundation

Deploy PostgreSQL as a StatefulSet with a headless Service and `volumeClaimTemplates`. Wait for the Pod to be Running and Ready, verify the PVC was auto-created, and test database connectivity.

Analogy🏏Cricket

🏏 Think of it like cricket: Step 1 is setting the batting lineup before the first ball. You're establishing the production baseline — 3 batsmen at the crease (3 replicas), with the team's fitness protocol in place (health probes), clear performance expectations (resource requests and limits). Everything that follows depends on this solid initial setup.

bash

cat > state/postgres-statefulset.yaml << 'EOF'
apiVersion: v1
kind: Service
metadata:
  name: postgres
  namespace: cricketpulse-db
spec:
  clusterIP: None
  selector: { app: postgres }
  ports: [{ name: postgres, port: 5432 }]
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  namespace: cricketpulse-db
spec:
  serviceName: postgres
  replicas: 1
  selector:
    matchLabels: { app: postgres }
  template:
    metadata:
      labels: { app: postgres }
    spec:
      containers:
        - name: postgres
          image: postgres:16-alpine
          ports: [{ containerPort: 5432, name: postgres }]
          envFrom:
            - secretRef: { name: postgres-secret }
          env:
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits: { cpu: 500m, memory: 512Mi }
          readinessProbe:
            exec:
              command: [pg_isready, -U, cricketer, -d, cricketpulse]
            initialDelaySeconds: 10
            periodSeconds: 5
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
    - metadata: { name: data }
      spec:
        storageClassName: standard
        accessModes: [ReadWriteOnce]
        resources:
          requests: { storage: 1Gi }
EOF

kubectl apply -f state/postgres-statefulset.yaml
kubectl rollout status statefulset/postgres -n cricketpulse-db

# Verify Pod and PVC
kubectl get pods,pvc -n cricketpulse-db
# Pod: postgres-0   1/1  Running
# PVC: data-postgres-0  Bound  1Gi

Step 2 — Core Logic

Write data to the database, delete the Pod (simulating a crash), watch it restart and remount the same PVC, and verify the data is still present. This is the core persistence guarantee that validates the StatefulSet and PVC design.

Analogy🏏Cricket

🏏 Think of it like cricket: Step 2 is the mid-series tactical review. The captain introduces a new player (v2 configuration) for one over to evaluate their performance before committing to a longer spell. The pause is the strategic timeout — the captain reviews the scoreboard for the new player's first over before deciding whether to continue the new approach or revert to the proven lineup.

bash

# Connect to the database and write test data
kubectl exec -it postgres-0 -n cricketpulse-db -- \
  psql -U cricketer -d cricketpulse -c "
    CREATE TABLE ipl_scores (
      id SERIAL PRIMARY KEY,
      match VARCHAR(100),
      team VARCHAR(50),
      score INTEGER,
      created_at TIMESTAMP DEFAULT NOW()
    );
    INSERT INTO ipl_scores (match, team, score)
    VALUES
      ('MI vs CSK Final', 'Mumbai Indians', 185),
      ('MI vs CSK Final', 'Chennai Super Kings', 179);
    SELECT * FROM ipl_scores;
  "
# Should show 2 rows

# DELETE the Pod (simulates a crash)
kubectl delete pod postgres-0 -n cricketpulse-db

# Watch the Pod restart automatically
kubectl get pods -n cricketpulse-db -w
# postgres-0   0/1   Terminating   → Pending → ContainerCreating → Running

# Verify the SAME PVC is remounted (data-postgres-0 is unchanged)
kubectl get pvc -n cricketpulse-db

# Check data is still there after restart
kubectl exec -it postgres-0 -n cricketpulse-db -- \
  psql -U cricketer -d cricketpulse -c 'SELECT * FROM ipl_scores;'
# 2 rows still present — data survived the Pod restart

Step 3 — Integration & Enhancement

Deploy a simple CricketPulse API application that connects to the database using the stable DNS name of the StatefulSet Pod. Verify the full data flow: API inserts data, API reads data, data is visible in the database directly.

Analogy🏏Cricket

🏏 Think of it like cricket: Step 3 is the batting collapse scenario — a rapid loss of wickets that requires the captain to call the team together, diagnose what went wrong, and switch back to a more conservative approach to save the innings. The bad deployment is the top-order collapse; the rollback is the captain's tactical reset. Every production engineer has this experience; the exercise ensures you've handled it in a controlled environment before it happens in production.

bash

# Create a ClusterIP Service for the database (for application access)
kubectl expose statefulset postgres \
  --name=postgres-svc \
  --port=5432 \
  --target-port=5432 \
  -n cricketpulse-db

# Verify DNS name works — exec into postgres-0 and query via Service
kubectl exec -it postgres-0 -n cricketpulse-db -- \
  psql -h postgres-svc -U cricketer -d cricketpulse -c 'SELECT count(*) FROM ipl_scores;'

# Verify stable Pod DNS name works
kubectl exec -it postgres-0 -n cricketpulse-db -- \
  psql -h postgres-0.postgres.cricketpulse-db.svc.cluster.local \
  -U cricketer -d cricketpulse -c 'SELECT version();'

# Run a debug pod in the same namespace to test cross-pod DNS
kubectl run pg-client -n cricketpulse-db \
  --image=postgres:16-alpine --rm -it -- \
  psql -h postgres-svc -U cricketer -d cricketpulse \
  -c 'SELECT * FROM ipl_scores;'

echo 'Connection via Service and stable Pod DNS confirmed'

Step 4 — Testing & Verification

Verify all StatefulSet guarantees and clean up carefully.

Analogy🏏Cricket

🏏 Think of it like cricket: Manually deleting a Pod and watching the Deployment replace it is like a captain testing his team's bench depth by pulling a fielder off mid-session and seeing the twelfth man walk on instantly, unprompted. Just as a well-run squad has a standing rule — 'the field always has eleven' — the Deployment holds a declared replica count and the controller reconciles reality back to it the moment a Pod vanishes. Just as changing the declared squad size (say to a five-a-side drill) makes the coach add or bench players to match, `kubectl scale` changes the desired count and the controller adds or removes Pods accordingly. The payoff: you see firsthand that Kubernetes is declarative — you state the number you want and the system continuously enforces it, healing deletions and honouring scale changes without manual babysitting.

bash

# Final verification
kubectl get statefulset,pods,pvc,svc -n cricketpulse-db

# Confirm PVC persists after scaling to 0
kubectl scale statefulset postgres -n cricketpulse-db --replicas=0
kubectl get pvc -n cricketpulse-db  # PVC still exists

# Scale back up — data still there
kubectl scale statefulset postgres -n cricketpulse-db --replicas=1
kubectl rollout status statefulset/postgres -n cricketpulse-db
kubectl exec -it postgres-0 -n cricketpulse-db -- \
  psql -U cricketer -d cricketpulse -c 'SELECT * FROM ipl_scores;'

# SAFE CLEANUP: scale to 0 first, then delete PVC manually
kubectl scale statefulset postgres -n cricketpulse-db --replicas=0
kubectl wait pod -l app=postgres -n cricketpulse-db --for=delete --timeout=60s
kubectl delete namespace cricketpulse-db
# Note: PVCs in the namespace are deleted with the namespace
echo 'State exercise complete!'

Warning: On kind clusters, the default `standard` StorageClass uses `rancher.io/local-path` which stores data in the Docker container representing the Kubernetes node. If the kind cluster is deleted (`kind delete cluster`), all PVC data is lost — the Docker container and its local storage are destroyed. This is expected for a local development environment. In production, always use a cloud StorageClass (EBS, GCE PD, Azure Disk) whose underlying storage outlives the cluster and its nodes.

Extension Challenge: Scale the StatefulSet to 3 replicas and observe that Pods start in order (0 must be Ready before 1 starts, 1 before 2). Verify each Pod has its own PVC (`data-postgres-0`, `data-postgres-1`, `data-postgres-2`). Then write data to postgres-0 directly, and verify that postgres-1 and postgres-2 (running independent PostgreSQL instances) do NOT have that data — confirming that PVCs are per-Pod and not shared. This demonstrates why database clustering protocols (streaming replication) are needed to synchronise data across StatefulSet replicas.

The `pg_isready` readiness probe is the correct probe for PostgreSQL — it returns success only when the database is accepting connections, not just when the process has started.
StatefulSet Pods restart with the exact same name and remount the same PVC — this is the core persistence guarantee; data survives Pod crashes and node rescheduling.
Stable DNS names (`postgres-0.postgres.<namespace>.svc.cluster.local`) work only when the headless Service name matches the StatefulSet's `serviceName:` field.
Expose a regular ClusterIP Service alongside the headless Service for application connections — applications use `postgres-svc` for general access and the Pod FQDN for specific Pod access.
Scale to 0 before deleting PVCs to ensure clean database shutdown — unclean shutdowns can corrupt PostgreSQL data files.
kind's local-path StorageClass is sufficient for development and learning but stores data in ephemeral Docker container storage — use cloud StorageClasses for any production or durable data requirements.