What You'll Build
In this exercise you will apply the complete M5 security hardening stack to the IPL analytics platform, transforming it from a functional but minimally secured deployment into a production-grade hardened system. You will create dedicated ServiceAccounts for each workload with precisely scoped RBAC Roles, apply Pod Security Admission enforcement at the `restricted` level to the production namespace, deploy the default-deny NetworkPolicy set with specific allow-rules for each service's actual traffic patterns, install Kyverno with the approved-registry and resource-limits validation policies, and run the complete security verification checklist that confirms each hardening layer works independently and does not break the platform's functionality. The exercise is structured to apply hardening in layers, verifying after each layer that the platform continues to serve correct responses — the operational discipline that prevents a security hardening exercise from becoming an unplanned outage.
Analogy🏏Cricket
🏏 Think of it like cricket: This lab's staging-then-production TLS pipeline is the ICC's pre-tour practice match protocol. Before the Test series proper, the touring team plays a two-day warm-up match against a local state side — not under ICC playing conditions, not with the official match balls, and not counted in official records. The warm-up match validates that the team's batting and bowling combinations work on local pitch conditions before committing to the conditions for the official five-day Test. The Let's Encrypt staging environment is that warm-up match: it issues real certificates signed by a staging CA (not trusted by browsers, like an unofficial match result), validates the complete ACME challenge flow, and confirms that DNS, network, and IAM configurations are correct — all without consuming the production rate limit. Switching to the production issuer is committing to the official Test: the certificate is now signed by Let's Encrypt's trusted CA (officially recognised), but any misconfiguration wastes an official certificate issuance. The certificate rotation simulation is the team testing their emergency substitution protocol — specifically waiting until the 'player' (certificate) is within the renewal window, confirming the automatic replacement process fires correctly, and verifying the new 'player' is ready to take the field. This reveals why the three-phase structure of the lab matters: each phase validates a distinct property of the TLS chain, and proceeding through them in order reduces the risk that a configuration problem discovered in production was one that staging testing would have caught.
Prerequisites
- The IPL analytics platform from M4 Exercise running with StatefulSet PostgreSQL, HPA on the API Deployment, and Ingress with TLS — all services responding correctly before beginning hardening.
- Kyverno installed via Helm as covered in Lesson 32 — verify with `kubectl get pods -n kyverno` showing all three Kyverno controller replicas Running.
- The production namespace does NOT yet have PSA enforce labels — this exercise adds them; verify with `kubectl get ns production --show-labels` confirming no `pod-security.kubernetes.io/enforce` label.
- A Calico or Cilium CNI installed — verify NetworkPolicy enforcement is active by creating two test Pods and confirming unrestricted connectivity before applying default-deny.
- The API endpoint accessible via `kubectl port-forward` or the Ingress URL for integration testing — each hardening step is followed by a connectivity verification against this endpoint.
Step 1 — Foundation
Apply RBAC hardening first — create dedicated ServiceAccounts and patch the default ServiceAccount to disable automount. RBAC hardening is the safest first step because it does not disrupt existing network connectivity or Pod creation; it only changes the API server credentials available to running Pods. Verify after applying that all existing Pods are still functional before proceeding to the next hardening layer.
Analogy🏏Cricket
🏏 Think of it like cricket: before the first official fixture, a serious team plays a full-dress practice match — real pitch, real umpires, real match conditions — knowing the result won't appear in any league table. Just as that practice match earns no points but proves the batting order, bowling plans, and fielding drills actually work under match pressure, the Let's Encrypt staging certificate won't be trusted by any browser but proves that the ACME challenge completes, DNS resolves correctly, and the IngressController routes traffic as designed. Just as a coach would never debut an untested game plan in a televised final, you never point at the production issuer first — a failed live attempt costs far more than a failed rehearsal. The payoff: every moving part of the certificate pipeline validated cheaply, so the later switch to production is a formality rather than a gamble.
bash
# Step 1: RBAC hardening
# Patch default ServiceAccount to disable automount in production
kubectl patch serviceaccount default -n production -p '{"automountServiceAccountToken": false}'
# Create purpose-specific ServiceAccounts
kubectl apply -f - << 'EOF'
apiVersion: v1
kind: ServiceAccount
metadata:
name: ipl-api-sa
namespace: production
automountServiceAccountToken: false # uses IRSA, no cluster API calls
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: ipl-postgres-sa
namespace: production
automountServiceAccountToken: false # postgres makes no API calls
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: ipl-redis-sa
namespace: production
automountServiceAccountToken: false # redis makes no API calls
EOF
# Update Deployments to use purpose-specific ServiceAccounts
kubectl patch deployment ipl-api -n production -p '{"spec":{"template":{"spec":{"serviceAccountName":"ipl-api-sa"}}}}'
kubectl patch statefulset ipl-postgres -n production -p '{"spec":{"template":{"spec":{"serviceAccountName":"ipl-postgres-sa"}}}}'
# Verify platform still works after RBAC change
kubectl port-forward service/ipl-api 8080:8000 -n production &
PF_PID=$!; sleep 3
STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health)
kill $PF_PID 2>/dev/null
echo "After RBAC hardening: HTTP $STATUS (expected 200)"
[ "$STATUS" = "200" ] && echo "PASS ✓ — platform unaffected by RBAC hardening" Step 2 — Core Logic
Apply PSA enforcement and NetworkPolicies as the second and third hardening layers. PSA enforcement is applied by first running a dry-run validation against all existing Pod specs to confirm they are `restricted`-compliant — if any fail, fix them before applying the label. NetworkPolicies are applied as a complete set starting with the default-deny and followed by all allow-rules simultaneously, to ensure the allow-rules are in place before the deny takes effect.
Analogy🏏Cricket
🏏 Think of it like cricket: once the practice match proves the plans work, the team steps into the official fixture — and everything must now count for real. Just as the practice-game scorecard is torn up so the official scorers start a fresh book, the staging certificate Secret is deleted so cert-manager issues a fresh production certificate rather than serving the old untrusted one. Just as the umpires formally verify the match ball and playing conditions before play begins, you verify the production certificate against the system certificate store — curl without -k must succeed. Then comes rehearsing the handover: just as a club renews a key player's contract well before it expires so there is never a day without him under contract, patching renewBefore to 89 days forces cert-manager to renew the 90-day certificate almost immediately, proving rotation happens automatically long before expiry. The payoff: trusted TLS in production plus demonstrated automatic renewal — no midnight expiry emergencies.
bash
# Step 2: PSA enforcement + NetworkPolicies
# ── PSA: dry-run validation before enabling enforcement ───────────────────
echo "Validating existing Pods against restricted profile..."
kubectl get pods -n production -o json | python3 -c "
import sys, json, subprocess
pods = json.load(sys.stdin)['items']
violations = []
for pod in pods:
result = subprocess.run(
['kubectl', 'auth', 'can-i', '--list'],
capture_output=True, text=True
)
# Simplified check: look for missing security context fields
spec = pod['spec']
for container in spec.get('containers', []):
sc = container.get('securityContext', {})
if not sc.get('runAsNonRoot'):
violations.append(f'{pod["metadata"]["name"]}/{container["name"]}: missing runAsNonRoot')
if not sc.get('allowPrivilegeEscalation') == False:
violations.append(f'{pod["metadata"]["name"]}/{container["name"]}: missing allowPrivilegeEscalation=false')
if violations:
print('VIOLATIONS FOUND — fix before enabling enforcement:')
for v in violations: print(f' {v}')
else:
print('All Pods compliant with restricted profile ✓')
"
# Apply PSA labels (assuming all Pods are compliant)
kubectl label namespace production pod-security.kubernetes.io/enforce=restricted pod-security.kubernetes.io/enforce-version=latest --overwrite
echo "PSA restricted enforcement enabled ✓"
# ── NetworkPolicies: apply complete set atomically ────────────────────────
kubectl apply -f - << 'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny-all, namespace: production}
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: allow-dns-egress, namespace: production}
spec:
podSelector: {}
policyTypes: [Egress]
egress:
- to:
- namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: kube-system}}
podSelector: {matchLabels: {k8s-app: kube-dns}}
ports: [{protocol: UDP, port: 53}, {protocol: TCP, port: 53}]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: allow-api-from-ingress, namespace: production}
spec:
podSelector: {matchLabels: {app: ipl-api}}
policyTypes: [Ingress]
ingress:
- from:
- namespaceSelector: {matchLabels: {kubernetes.io/metadata.name: ingress-nginx}}
podSelector: {matchLabels: {app.kubernetes.io/name: ingress-nginx}}
ports: [{port: 8000, protocol: TCP}]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: allow-api-to-data, namespace: production}
spec:
podSelector: {matchLabels: {app: ipl-api}}
policyTypes: [Egress]
egress:
- to: [{podSelector: {matchLabels: {app: ipl-postgres}}}]
ports: [{port: 5432, protocol: TCP}]
- to: [{podSelector: {matchLabels: {app: ipl-redis}}}]
ports: [{port: 6379, protocol: TCP}]
- to: [{ipBlock: {cidr: 0.0.0.0/0, except: [10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16]}}]
ports: [{port: 443, protocol: TCP}]
EOF
echo "NetworkPolicies applied ✓"
# Verify API still accessible after NetworkPolicy application
kubectl port-forward service/ipl-api 8080:8000 -n production &
PF_PID=$!; sleep 3
STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/health)
kill $PF_PID 2>/dev/null
echo "After NetworkPolicy: HTTP $STATUS (expected 200)" Step 3 — Integration & Enhancement
Apply Kyverno policies as the final hardening layer. Install the approved-registry and resource-limits policies in Audit mode first, verify zero violations in the PolicyReport, then switch to Enforce mode. Run the complete security verification checklist confirming all four hardening layers are active and the platform serves correct responses.
Analogy🏏Cricket
🏏 Think of it like cricket: on the eve of a tournament the venue runs a full walk-through — a spectator's journey from the car park, through the ticket gate, to the correct stand, with stewards redirecting anyone who wanders toward the wrong entrance. Just as that walk-through exercises every link in the chain, the end-to-end check verifies DNS resolution, the TLS handshake with the production certificate, routing to the correct backend service, and the HTTPS redirect that steers plain-HTTP stragglers to the secure entrance. Just as gate staff read the stand printed on each ticket and route spectators accordingly, host-based routing reads the hostname and sends pgAdmin traffic to its own backend while API traffic goes to the API. And just as an all-venue tournament pass admits its holder to every ground without a separate ticket per stadium, the wildcard certificate covers both subdomains with a single credential. The payoff: one entry point, correctly securing and routing every kind of visitor.
bash
# Step 3: Kyverno policies + final verification checklist
# Apply Kyverno policies in Audit mode
kubectl apply -f - << 'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-approved-registry
spec:
validationFailureAction: Audit # start in Audit, switch to Enforce after review
background: true
rules:
- name: check-registry
match: {any: [{resources: {kinds: [Pod], namespaces: [production]}}]}
validate:
message: "Images must be from approved ECR registry"
pattern:
spec:
containers:
- image: "123456789.dkr.ecr.ap-south-1.amazonaws.com/*|(postgres|redis|alpine|busybox):*"
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-resource-limits
spec:
validationFailureAction: Audit
background: true
rules:
- name: check-limits
match: {any: [{resources: {kinds: [Deployment, StatefulSet], namespaces: [production]}}]}
validate:
message: "All containers must have cpu and memory limits"
pattern:
spec:
template:
spec:
containers:
- resources:
limits:
memory: "?*"
cpu: "?*"
EOF
sleep 30 # wait for Kyverno to evaluate existing resources
# Check PolicyReport for violations
echo "=== Kyverno PolicyReport ==="
kubectl get policyreport -n production -o jsonpath='{range .items[*]}{.metadata.name}: pass={.summary.pass} fail={.summary.fail}{"
"}{end}'
# If fail=0, switch to Enforce mode
kubectl patch clusterpolicy require-approved-registry -p '{"spec":{"validationFailureAction":"Enforce"}}' --type merge
kubectl patch clusterpolicy require-resource-limits -p '{"spec":{"validationFailureAction":"Enforce"}}' --type merge
echo ""
echo "=== Final Security Hardening Verification ==="
# 1. RBAC: default SA has automount disabled
echo -n "1. Default SA automount disabled: "
AUTOMOUNT=$(kubectl get serviceaccount default -n production -o jsonpath='{.automountServiceAccountToken}')
[ "$AUTOMOUNT" = "false" ] && echo "PASS ✓" || echo "FAIL ✗"
# 2. PSA: enforce=restricted label present
echo -n "2. PSA restricted enforcement: "
PSA=$(kubectl get ns production -o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce}')
[ "$PSA" = "restricted" ] && echo "PASS ✓" || echo "FAIL ✗ (got: $PSA)"
# 3. NetworkPolicy: default-deny-all exists
echo -n "3. Default-deny NetworkPolicy: "
kubectl get networkpolicy default-deny-all -n production > /dev/null 2>&1 && echo "PASS ✓" || echo "FAIL ✗"
# 4. Kyverno: policies in Enforce mode
echo -n "4. Kyverno enforce mode active: "
ACTION=$(kubectl get clusterpolicy require-approved-registry -o jsonpath='{.spec.validationFailureAction}')
[ "$ACTION" = "Enforce" ] && echo "PASS ✓" || echo "FAIL ✗"
# 5. Platform functional
echo -n "5. API responds correctly: "
kubectl port-forward service/ipl-api 8080:8000 -n production &
PF_PID=$!; sleep 3
STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/batters/rohit_sharma)
kill $PF_PID 2>/dev/null
[ "$STATUS" = "200" ] && echo "PASS ✓" || echo "FAIL ✗ (HTTP $STATUS)"
echo "=== All hardening layers applied and verified ===" Warning: Never apply the default-deny NetworkPolicy before creating the DNS allow-rule and all service-specific allow-rules. Applying default-deny-all first and then adding allow-rules one at a time will create a window where DNS resolution is broken (causing all hostname lookups to fail), PostgreSQL is unreachable (causing API errors), and Redis is unreachable (causing cache failures) — each step takes a few seconds, but if any step fails or is interrupted, the platform is left in a partially-secured, partially-broken state. Apply all NetworkPolicies in a single `kubectl apply -f netpolicies/` directory command or a single multi-document YAML file to ensure the default-deny and all allow-rules are created atomically in a single API call batch.
Extension Challenge: Integrate the Vault Agent sidecar with the PostgreSQL StatefulSet to replace the static database password Secret with Vault-generated dynamic credentials. Configure the Vault Kubernetes auth method to accept the `ipl-postgres-sa` ServiceAccount, create a Vault database role with CREATE/GRANT permissions for schema initialization, and add the Vault Agent annotations to the API Deployment to render the database URL from Vault at startup. This extension exercises all three components of the secrets management stack: RBAC ServiceAccount, Vault Kubernetes auth, and Vault database secrets engine working together.
- Apply hardening layers sequentially with verification after each step — RBAC first (zero disruption risk), PSA second (dry-run validation required), NetworkPolicies third (apply all rules atomically), Kyverno fourth (audit mode first).
- PSA enforcement on a running namespace requires verifying all existing Pods are compliant via dry-run before applying the label — non-compliant running Pods continue until their next recreation, at which point they will be blocked.
- Apply NetworkPolicies as a complete set (default-deny + all allow-rules) in a single kubectl apply command to avoid a window where default-deny is active without the DNS and service-specific allow-rules.
- Kyverno policies should always start in Audit mode, with PolicyReport review confirming zero violations before switching to Enforce — never apply Enforce directly without an audit phase.
- Verify platform functionality after each hardening layer by testing the API health endpoint — a passing test after each layer confirms the hardening was applied correctly without breaking the application.
- The complete hardening stack — RBAC least-privilege, PSA restricted, NetworkPolicy default-deny, Kyverno governance — addresses four independent threat categories: API server compromise, container escape, lateral movement, and supply chain.