Scaling Practice — Load-Driven Autoscaling

What You'll Build

You will configure the Horizontal Pod Autoscaler for the CricketPulse API, install metrics-server to enable CPU-based scaling, generate artificial load to trigger scale-up events, observe the HPA in action, and verify that the Pod count increases automatically. You will then stop the load, observe the scale-down cooldown period, and verify the fleet returns to minimum replicas. You will also configure a Pod Disruption Budget and verify it correctly restricts node drain operations.

Analogy🏏Cricket

🏏 Think of it like cricket: This exercise is a complete match simulation — not just batting practice. You'll face the full sequence: opening innings (initial deployment), the change of tactics mid-game (rolling update with canary check), an unexpected top-order collapse (bad deployment), and the team's recovery (rollback). Each scenario tests a different aspect of your Kubernetes deployment skills. Just as a batsman who's only practiced in nets needs match experience to handle the unexpected pressure of a live game, you need to experience deployment, update, failure, and recovery in sequence to build the operational instincts for real production management.

Prerequisites

A running kind cluster from Lesson 4 (or recreated fresh).
kubectl configured and pointing to the cluster.
Lessons 17–19 completed — understanding of HPA, probes, and resource requests.
Apache Bench (`ab`) or `hey` load testing tool installed — `brew install hey` or `apt install apache2-utils`.
Understanding that metrics-server requires a small installation step on kind clusters.

Setup & Project Structure

Create the scaling exercise namespace, install metrics-server (required for CPU-based HPA), and deploy the CricketPulse API with proper resource requests to enable HPA operation.

Analogy🏏Cricket

🏏 Think of it like cricket: Simulating different CricketPulse API versions with a ConfigMap and stock nginx is like rehearsing a batting-order change using the same players wearing different bib numbers, instead of recruiting brand-new specialists for every drill. Just as swapping a player's instructions on a laminated card lets the coach test how the side responds to a new plan without signing anyone, injecting different response text through a ConfigMap lets nginx serve 'v1' or 'v2' without building a custom Docker image. Just as a good academy sets out a clear practice plan — where each drill lives, which card belongs to which scenario — you lay out a tidy manifest directory before starting. The payoff: you can rehearse realistic multi-version rollout behaviour quickly and cheaply, learning the mechanics without the overhead of building and pushing real container images.

bash

mkdir -p cricketpulse-k8s/scaling
cd cricketpulse-k8s

# Create namespace
kubectl create namespace cricketpulse-scaling

# Install metrics-server (kind requires --kubelet-insecure-tls flag)
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# Patch metrics-server for kind (self-signed certificates)
kubectl patch deployment metrics-server \
  -n kube-system \
  --type='json' \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

# Wait for metrics-server to be ready
kubectl wait deployment -n kube-system metrics-server \
  --for=condition=Available --timeout=60s

# Verify metrics-server works
kubectl top nodes
# NAME                             CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
# cricketpulse-cluster-worker      25m          0%     450Mi           11%

echo 'Metrics-server ready'

Step 1 — Foundation

Deploy CricketPulse API with a resource-defined container, create the HPA, and verify the initial state. The HPA will show 'unknown' metrics initially while metrics-server warms up.

Analogy🏏Cricket

🏏 Think of it like cricket: Step 1 is setting the batting lineup before the first ball. You're establishing the production baseline — 3 batsmen at the crease (3 replicas), with the team's fitness protocol in place (health probes), clear performance expectations (resource requests and limits). Everything that follows depends on this solid initial setup.

bash

# Deploy CricketPulse API with resource requests (required for HPA)
cat > scaling/deployment.yaml << 'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cricketpulse-api
  namespace: cricketpulse-scaling
spec:
  replicas: 2
  selector:
    matchLabels: { app: cricketpulse-api }
  template:
    metadata:
      labels: { app: cricketpulse-api }
    spec:
      containers:
        - name: api
          image: nginx:alpine
          ports: [{ containerPort: 80 }]
          resources:
            requests: { cpu: 100m, memory: 64Mi }   # ← Required for HPA
            limits:   { cpu: 300m, memory: 128Mi }
EOF

# ClusterIP Service for load testing via port-forward
kubectl expose deployment cricketpulse-api \
  --port=80 --target-port=80 \
  -n cricketpulse-scaling

# Create the HPA
kubectl autoscale deployment cricketpulse-api \
  --min=2 --max=10 --cpu-percent=50 \
  -n cricketpulse-scaling

# Verify HPA is configured (may show <unknown> briefly)
kubectl get hpa -n cricketpulse-scaling
# Wait 60 seconds then check again — should show actual CPU%

Step 2 — Core Logic

Generate load against the CricketPulse API to push CPU above the 50% target and trigger the HPA to scale up. Watch the HPA respond in real time.

Analogy🏏Cricket

🏏 Think of it like cricket: Step 2 is the mid-series tactical review. The captain introduces a new player (v2 configuration) for one over to evaluate their performance before committing to a longer spell. The pause is the strategic timeout — the captain reviews the scoreboard for the new player's first over before deciding whether to continue the new approach or revert to the proven lineup.

bash

# Port-forward to enable load testing
kubectl port-forward svc/cricketpulse-api 8080:80 \
  -n cricketpulse-scaling &
PF_PID=$!
sleep 2

# Verify port-forward works
curl -s http://localhost:8080/ | head -1

# Generate sustained load in background to trigger HPA
# Method 1: using hey (if installed)
hey -z 120s -c 10 http://localhost:8080/ &
LOAD_PID=$!

# Method 2: using ab (apache bench alternative)
# ab -t 120 -c 10 http://localhost:8080/ &

# Method 3: pure bash if neither tool is available
# for i in $(seq 1 200); do curl -s http://localhost:8080/ >/dev/null & done

# Watch HPA react in real time (every 15 seconds)
echo 'Watching HPA scale up...'
for i in $(seq 1 20); do
  echo '--- Iteration '$i' ---'
  kubectl get hpa,pods -n cricketpulse-scaling 2>&1 | \
    grep -E 'NAME|api|READY|HPA'
  sleep 15
done

# Stop load
kill $LOAD_PID 2>/dev/null
kill $PF_PID 2>/dev/null
echo 'Load stopped — watching scale-down...'

Step 3 — Integration & Enhancement

Observe the scale-down cooldown (HPA waits 5 minutes before reducing replicas), then create a Pod Disruption Budget and verify it correctly restricts the number of Pods that can be disrupted simultaneously.

Analogy🏏Cricket

🏏 Think of it like cricket: Step 3 is the batting collapse scenario — a rapid loss of wickets that requires the captain to call the team together, diagnose what went wrong, and switch back to a more conservative approach to save the innings. The bad deployment is the top-order collapse; the rollback is the captain's tactical reset. Every production engineer has this experience; the exercise ensures you've handled it in a controlled environment before it happens in production.

bash

# Watch scale-down (happens after 5 minute stabilisation window)
echo 'Load stopped. Scale-down will happen after ~5 minutes...'
for i in $(seq 1 25); do
  echo "$(date +%H:%M:%S): $(kubectl get deployment cricketpulse-api \
    -n cricketpulse-scaling -o jsonpath='{.spec.replicas}') replicas"
  sleep 30
done

# Create a Pod Disruption Budget
cat > scaling/pdb.yaml << 'EOF'
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: cricketpulse-api-pdb
  namespace: cricketpulse-scaling
spec:
  minAvailable: 2
  selector:
    matchLabels: { app: cricketpulse-api }
EOF
kubectl apply -f scaling/pdb.yaml

# Verify PDB status
kubectl get pdb -n cricketpulse-scaling
# ALLOWED DISRUPTIONS should be: total replicas - minAvailable
# With 3 replicas and minAvailable 2: 1 allowed disruption
# With 2 replicas and minAvailable 2: 0 allowed disruptions

kubectl describe pdb cricketpulse-api-pdb -n cricketpulse-scaling

Step 4 — Testing & Verification

Complete verification and cleanup.

Analogy🏏Cricket

🏏 Think of it like cricket: Manually deleting a Pod and watching the Deployment replace it is like a captain testing his team's bench depth by pulling a fielder off mid-session and seeing the twelfth man walk on instantly, unprompted. Just as a well-run squad has a standing rule — 'the field always has eleven' — the Deployment holds a declared replica count and the controller reconciles reality back to it the moment a Pod vanishes. Just as changing the declared squad size (say to a five-a-side drill) makes the coach add or bench players to match, `kubectl scale` changes the desired count and the controller adds or removes Pods accordingly. The payoff: you see firsthand that Kubernetes is declarative — you state the number you want and the system continuously enforces it, healing deletions and honouring scale changes without manual babysitting.

bash

# Final state check
kubectl get hpa,deployment,pods,pdb -n cricketpulse-scaling

# Verify HPA history (should show scale events)
kubectl describe hpa -n cricketpulse-scaling | grep -A 20 Events
# Normal  SuccessfulRescale  Scaled up to N replicas
# Normal  SuccessfulRescale  Scaled down to 2 replicas

# Confirm QoS class of running Pods
POD=$(kubectl get pods -n cricketpulse-scaling \
  -o jsonpath='{.items[0].metadata.name}')
kubectl get pod $POD -n cricketpulse-scaling \
  -o jsonpath='{.status.qosClass}'
# Should be: Burstable (requests < limits)

# Check resource consumption vs requests
kubectl top pods -n cricketpulse-scaling

# Cleanup
kubectl delete namespace cricketpulse-scaling
echo 'Scaling exercise complete!'

Warning: metrics-server may take 60–90 seconds after installation before it can provide CPU metrics to the HPA. During this time, `kubectl get hpa` shows `<unknown>/50%` for the CPU target. This is normal — wait for a complete minute, then run `kubectl top nodes` to confirm metrics-server is working before starting the load test. If metrics-server never starts providing metrics, check its logs: `kubectl logs -n kube-system -l k8s-app=metrics-server`.

Extension Challenge: After completing the exercise, apply a ResourceQuota to the namespace with `requests.cpu: 500m` and try to scale the Deployment to 10 replicas (each requesting 100m CPU = 1000m total). Observe the ResourceQuota blocking the scale-up: `kubectl describe resourcequota` shows the quota exceeded error. Then increase the quota to `2` CPU and verify the scale-up succeeds. This demonstrates how ResourceQuotas provide a safety net against unbounded resource consumption.

HPA requires metrics-server installed in the cluster and resource requests set on containers to calculate CPU utilisation percentage — without either, HPA shows `<unknown>` metrics.
Scale-up happens quickly (within 15–30 seconds of metrics showing above target); scale-down is delayed by the stabilisation window (default 5 minutes) to prevent oscillation.
PDB `ALLOWED DISRUPTIONS` shows how many Pods can currently be disrupted — this value decreases as replica count decreases, blocking node drains when at minimum.
Use `kubectl describe hpa` to see scaling event history — this is invaluable for debugging why the HPA did or didn't scale at an expected time.
The HPA-managed `spec.replicas` field is overwritten by the HPA on every evaluation cycle — never manually set replicas on HPA-managed Deployments.
`kubectl top pods` shows current resource consumption vs requests — use this to verify that CPU requests are appropriately sized (not drastically over- or under-estimated).