Lab — Kyverno policies for supply chain enforcement and security governance

What You'll Build

In this lab you will implement a comprehensive Kyverno policy suite that enforces supply chain integrity and security governance across the entire IPL analytics platform, building on the Cosign signing pipeline from M1 Lab 7 and the Kyverno installation from M5 Exercise. The policy suite covers four enforcement domains: supply chain integrity (Cosign signature verification that blocks deployment of unsigned images), security hardening (PSA-complementary policies for resource limits, seccomp profiles, and non-root user requirements), operational governance (required labels, approved StorageClasses, and Ingress TLS enforcement), and automated remediation (mutate policies that inject missing standard labels and generate ResourceQuotas for new namespaces). By the end of the lab, attempting to deploy an unsigned image, a privileged container, an Ingress without TLS, or a namespace without a ResourceQuota will all be blocked by the policy engine with descriptive error messages.

The lab is designed to demonstrate that policy-as-code enforcement is not merely additive security but architecturally transformative: instead of relying on documentation, code review, and post-deployment audits to ensure every team follows the security standard, the cluster's admission layer enforces the standard technically. A developer who forgets to add resource limits to their Deployment receives an immediate, descriptive rejection rather than discovering the omission three weeks later in a capacity incident. An engineer who tries to deploy an unsigned image receives a Kyverno rejection message identifying the exact missing control rather than a security audit finding months after deployment. This shift from detective to preventive controls is the primary operational benefit of policy-as-code governance.

Analogy🏏Cricket

🏏 Think of it like cricket: This lab's staging-then-production TLS pipeline is the ICC's pre-tour practice match protocol. Before the Test series proper, the touring team plays a two-day warm-up match against a local state side — not under ICC playing conditions, not with the official match balls, and not counted in official records. The warm-up match validates that the team's batting and bowling combinations work on local pitch conditions before committing to the conditions for the official five-day Test. The Let's Encrypt staging environment is that warm-up match: it issues real certificates signed by a staging CA (not trusted by browsers, like an unofficial match result), validates the complete ACME challenge flow, and confirms that DNS, network, and IAM configurations are correct — all without consuming the production rate limit. Switching to the production issuer is committing to the official Test: the certificate is now signed by Let's Encrypt's trusted CA (officially recognised), but any misconfiguration wastes an official certificate issuance. The certificate rotation simulation is the team testing their emergency substitution protocol — specifically waiting until the 'player' (certificate) is within the renewal window, confirming the automatic replacement process fires correctly, and verifying the new 'player' is ready to take the field. This reveals why the three-phase structure of the lab matters: each phase validates a distinct property of the TLS chain, and proceeding through them in order reduces the risk that a configuration problem discovered in production was one that staging testing would have caught.

Prerequisites

Kyverno v1.11+ installed with three replicas in the `kyverno` namespace — verify with `kubectl get pods -n kyverno` and confirm all pods are Running before beginning.
The IPL scorecard image signed with Cosign and pushed to ECR from M1 Lab 7 — the supply chain policy verifies the signature; an unsigned image in the registry will cause the first test to fail.
The M5 Exercise hardening applied to the production namespace — the lab adds policy coverage on top of the existing RBAC, PSA, and NetworkPolicy layers.
The `kubectl kyverno` CLI plugin installed locally for policy testing — `kubectl kyverno version` should return v1.11+ before writing any policy.
A second image tag that is intentionally unsigned prepared in ECR — used to verify that the supply chain policy correctly blocks unsigned images while admitting signed ones.

Setup & Project Structure

Organise the policy suite in a `k8s/policies/` directory with subdirectories for each domain: `supply-chain/`, `security/`, `governance/`, and `remediation/`. Each policy file should contain exactly one ClusterPolicy and its associated test cases, enabling per-policy CI testing and per-policy rollout in Audit mode before Enforce. Structure the file names to reflect the enforcement mode: `supply-chain-enforce.yaml` for Enforce policies, `governance-audit.yaml` for Audit-only policies that generate advisory reports.

Analogy🏏Cricket

🏏 Think of it like cricket: setting up a tournament venue is not one job but a strict sequence of jobs, and smart boards hire a professional event crew instead of doing each task by hand. Just as the event crew handles seating, accreditation desks, and broadcast rigging as one coordinated package, Helm installs the Nginx IngressController and cert-manager with all their CRDs, ServiceAccounts, and RBAC in one release instead of dozens of hand-applied manifests. And order matters: just as the DRS cameras and replay screens must be rigged and tested before the third umpire takes his seat — an official with no working replay feed is useless and the review system collapses — cert-manager's CRDs must exist before the controller starts, or it crashes on startup before its CRD admission webhooks ever come online. The payoff: a correctly sequenced, fully provisioned venue — controller infrastructure that comes up cleanly the first time.

bash

# Project structure for the policy suite

mkdir -p k8s/policies/{supply-chain,security,governance,remediation}
mkdir -p k8s/policies/tests/{supply-chain,security,governance}

# Verify Kyverno is ready
kubectl get deployment kyverno -n kyverno
# NAME      READY   UP-TO-DATE   AVAILABLE
# kyverno   3/3     3            3        ← three replicas for HA

# Verify the signed image exists in ECR
aws ecr describe-images   --repository-name ipl-scorecard   --image-ids imageTag=latest   --region ap-south-1   --query 'imageDetails[0].[imageTags,imagePushedAt]'

# Confirm signature is present
cosign tree 123456789.dkr.ecr.ap-south-1.amazonaws.com/ipl-scorecard:latest   | grep "sig"
# Expected: signature artifact in the tree ✓

# Prepare unsigned test image (use a public image from Docker Hub to test blocking)
docker pull python:3.11-slim
docker tag python:3.11-slim   123456789.dkr.ecr.ap-south-1.amazonaws.com/ipl-scorecard:unsigned-test
docker push   123456789.dkr.ecr.ap-south-1.amazonaws.com/ipl-scorecard:unsigned-test
# This image has NO Cosign signature — supply chain policy must block it

echo "Setup complete. Policy suite ready to write."
echo "Directory structure:"
tree k8s/policies/

Step 1 — Foundation

Write the supply chain policy that verifies Cosign signatures at admission time, test it locally against signed and unsigned test manifests, then deploy it in Audit mode and verify the PolicyReport shows zero violations for existing resources before switching to Enforce.

Analogy🏏Cricket

🏏 Think of it like cricket: before the first official fixture, a serious team plays a full-dress practice match — real pitch, real umpires, real match conditions — knowing the result won't appear in any league table. Just as that practice match earns no points but proves the batting order, bowling plans, and fielding drills actually work under match pressure, the Let's Encrypt staging certificate won't be trusted by any browser but proves that the ACME challenge completes, DNS resolves correctly, and the IngressController routes traffic as designed. Just as a coach would never debut an untested game plan in a televised final, you never point at the production issuer first — a failed live attempt costs far more than a failed rehearsal. The payoff: every moving part of the certificate pipeline validated cheaply, so the later switch to production is a formality rather than a gamble.

yaml

# k8s/policies/supply-chain/require-cosign-signature.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-cosign-signature
  annotations:
    policies.kyverno.io/title: Require Cosign Image Signature
    policies.kyverno.io/description: >-
      All images deployed to production must be signed by the
      organisation's CI pipeline using Cosign keyless signing.
      This prevents deployment of tampered or unverified images.
spec:
  validationFailureAction: Enforce
  background: false   # only evaluate at admission, not for existing resources
  rules:
    - name: check-image-signature
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [production]
      verifyImages:
        - imageReferences:
            - "123456789.dkr.ecr.ap-south-1.amazonaws.com/ipl-scorecard:*"
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/org/ipl-scorecard/.github/workflows/build-scan-push.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev
          # Also verify SLSA provenance attestation is present
          attestations:
            - predicateType: https://slsa.dev/provenance/v0.2

---
# Local test before deploying
# k8s/policies/tests/supply-chain/signed-pod.yaml (should PASS)
apiVersion: v1
kind: Pod
metadata:
  name: signed-pod
  namespace: production
spec:
  containers:
    - name: app
      image: 123456789.dkr.ecr.ap-south-1.amazonaws.com/ipl-scorecard:latest
      securityContext: {runAsNonRoot: true, runAsUser: 10001, allowPrivilegeEscalation: false, capabilities: {drop: [ALL]}}
      resources: {requests: {cpu: 100m, memory: 128Mi}, limits: {cpu: 500m, memory: 256Mi}}

# k8s/policies/tests/supply-chain/unsigned-pod.yaml (should FAIL)
apiVersion: v1
kind: Pod
metadata:
  name: unsigned-pod
  namespace: production
spec:
  containers:
    - name: app
      image: 123456789.dkr.ecr.ap-south-1.amazonaws.com/ipl-scorecard:unsigned-test

Step 2 — Core Logic

Deploy the security hardening and operational governance policies. The security policies add defence-in-depth on top of PSA enforcement — specifically requiring seccomp profiles (which PSA requires but does not validate the type), and requiring explicit non-root user IDs (PSA requires `runAsNonRoot: true` but allows any non-zero UID). The governance policies enforce operational standards that PSA and NetworkPolicies do not cover: Ingress TLS requirement, approved StorageClass restriction, and required pod labels for observability.

Analogy🏏Cricket

🏏 Think of it like cricket: once the practice match proves the plans work, the team steps into the official fixture — and everything must now count for real. Just as the practice-game scorecard is torn up so the official scorers start a fresh book, the staging certificate Secret is deleted so cert-manager issues a fresh production certificate rather than serving the old untrusted one. Just as the umpires formally verify the match ball and playing conditions before play begins, you verify the production certificate against the system certificate store — curl without -k must succeed. Then comes rehearsing the handover: just as a club renews a key player's contract well before it expires so there is never a day without him under contract, patching renewBefore to 89 days forces cert-manager to renew the 90-day certificate almost immediately, proving rotation happens automatically long before expiry. The payoff: trusted TLS in production plus demonstrated automatic renewal — no midnight expiry emergencies.

yaml

# Security + governance policies

# k8s/policies/security/require-seccomp.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-runtime-default-seccomp
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-seccomp-profile
      match: {any: [{resources: {kinds: [Pod], namespaces: [production]}}]}
      validate:
        message: "Pods must have seccompProfile.type RuntimeDefault or Localhost"
        pattern:
          spec:
            securityContext:
              seccompProfile:
                type: "RuntimeDefault | Localhost"

---
# k8s/policies/governance/require-ingress-tls.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-ingress-tls
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-ingress-tls
      match: {any: [{resources: {kinds: [Ingress], namespaces: [production]}}]}
      validate:
        message: "All production Ingress resources must configure TLS."
        deny:
          conditions:
            any:
              - key: "{{ request.object.spec.tls[] | length(@) }}"
                operator: LessThan
                value: 1

---
# k8s/policies/governance/require-approved-storageclass.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-approved-storageclass
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: check-storageclass
      match: {any: [{resources: {kinds: [PersistentVolumeClaim], namespaces: [production]}}]}
      validate:
        message: "PVCs in production must use 'gp3-encrypted' StorageClass."
        pattern:
          spec:
            storageClassName: "gp3-encrypted"

---
# k8s/policies/remediation/inject-standard-labels.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: inject-standard-labels
spec:
  rules:
    - name: add-env-label
      match: {any: [{resources: {kinds: [Deployment, StatefulSet], namespaces: [production, staging]}}]}
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              environment: "{{request.object.metadata.namespace}}"
              managed-by: +(kyverno-injected)

---
# k8s/policies/remediation/generate-ns-quota.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: generate-namespace-quota
spec:
  rules:
    - name: create-default-quota
      match:
        any:
          - resources:
              kinds: [Namespace]
              selector:
                matchLabels:
                  environment: production
      generate:
        apiVersion: v1
        kind: ResourceQuota
        name: default-quota
        namespace: "{{request.object.metadata.name}}"
        synchronize: true
        data:
          spec:
            hard:
              requests.cpu: "4"
              requests.memory: 8Gi
              limits.cpu: "8"
              limits.memory: 16Gi
              count/pods: "30"

Step 3 — Integration & Enhancement

Deploy all policies, run the complete test suite verifying each policy blocks non-compliant resources and admits compliant ones, and produce the final security posture dashboard that shows all active policies, their enforcement modes, and the current PolicyReport status across all namespaces.

Analogy🏏Cricket

🏏 Think of it like cricket: on the eve of a tournament the venue runs a full walk-through — a spectator's journey from the car park, through the ticket gate, to the correct stand, with stewards redirecting anyone who wanders toward the wrong entrance. Just as that walk-through exercises every link in the chain, the end-to-end check verifies DNS resolution, the TLS handshake with the production certificate, routing to the correct backend service, and the HTTPS redirect that steers plain-HTTP stragglers to the secure entrance. Just as gate staff read the stand printed on each ticket and route spectators accordingly, host-based routing reads the hostname and sends pgAdmin traffic to its own backend while API traffic goes to the API. And just as an all-venue tournament pass admits its holder to every ground without a separate ticket per stadium, the wildcard certificate covers both subdomains with a single credential. The payoff: one entry point, correctly securing and routing every kind of visitor.

bash

# Deploy all policies and run verification test suite

kubectl apply -f k8s/policies/supply-chain/
kubectl apply -f k8s/policies/security/
kubectl apply -f k8s/policies/governance/
kubectl apply -f k8s/policies/remediation/

sleep 15  # allow Kyverno to load and activate all policies

echo "=== M5 Lab — Policy Suite Verification ==="

# ── Test 1: Supply chain — signed image admitted ───────────────────────────
echo -n "1. Signed image admitted:              "
kubectl run supply-chain-signed-test   --image=123456789.dkr.ecr.ap-south-1.amazonaws.com/ipl-scorecard:latest   --dry-run=server -n production > /dev/null 2>&1   && echo "PASS ✓" || echo "FAIL ✗"

# ── Test 2: Supply chain — unsigned image blocked ──────────────────────────
echo -n "2. Unsigned image blocked:             "
kubectl run supply-chain-unsigned-test   --image=123456789.dkr.ecr.ap-south-1.amazonaws.com/ipl-scorecard:unsigned-test   --dry-run=server -n production 2>&1 | grep -q "image verification failed"   && echo "PASS ✓" || echo "FAIL ✗"

# ── Test 3: Security — Pod without seccomp blocked ─────────────────────────
echo -n "3. Pod without seccomp blocked:        "
kubectl apply --dry-run=server -f - << 'EOF' 2>&1 | grep -q "seccompProfile"   && echo "PASS ✓" || echo "FAIL ✗"
apiVersion: v1
kind: Pod
metadata: {name: no-seccomp-test, namespace: production}
spec:
  containers:
    - name: app
      image: 123456789.dkr.ecr.ap-south-1.amazonaws.com/ipl-scorecard:latest
      securityContext: {runAsNonRoot: true, runAsUser: 10001, allowPrivilegeEscalation: false, capabilities: {drop: [ALL]}}
      # MISSING: seccompProfile — should be blocked by require-runtime-default-seccomp
EOF

# ── Test 4: Governance — Ingress without TLS blocked ─────────────────────
echo -n "4. Ingress without TLS blocked:        "
kubectl apply --dry-run=server -f - << 'EOF' 2>&1 | grep -q "TLS"   && echo "PASS ✓" || echo "FAIL ✗"
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: {name: no-tls-test, namespace: production}
spec:
  rules: [{host: test.example.com, http: {paths: [{path: /, pathType: Prefix, backend: {service: {name: ipl-api, port: {number: 8000}}}}]}}]
  # MISSING: spec.tls — should be blocked
EOF

# ── Test 5: PolicyReport shows zero fails for existing resources ───────────
echo -n "5. Zero policy violations existing:    "
FAILS=$(kubectl get policyreport -n production   -o jsonpath='{.items[*].summary.fail}' | tr ' ' '
' | paste -sd+ | bc 2>/dev/null || echo "0")
[ "${FAILS:-0}" -eq "0" ] && echo "PASS ✓" || echo "FAIL ✗ ($FAILS violations)"

# ── Security posture dashboard ─────────────────────────────────────────────
echo ""
echo "=== Active Policy Suite ==="
kubectl get clusterpolicy -o custom-columns="NAME:.metadata.name,ACTION:.spec.validationFailureAction,BACKGROUND:.spec.background"   | sort

echo ""
echo "=== PolicyReport Summary ==="
kubectl get policyreport -A   -o custom-columns="NAMESPACE:.metadata.namespace,PASS:.summary.pass,FAIL:.summary.fail,WARN:.summary.warn"

echo ""
echo "=== Platform security posture: HARDENED ==="

Step 4 — Testing & Verification

Complete the final lab checklist and produce a comprehensive security posture summary that maps each implemented control to the threat category it addresses. This mapping makes the security architecture visible as an integrated system rather than a collection of independently-applied tools, enabling clear communication of the security model to stakeholders, auditors, and new team members.

Analogy🏏Cricket

🏏 Think of it like cricket: at the midpoint of a season, a good head coach doesn't just check the points table — he traces how every department produced it: the academy that developed the players, the selectors who picked the squad, and the matchday operations that got the team onto the field. Just as that review makes the whole club visible as one system rather than isolated departments, the M1–M4 architecture summary traces each component's role — the container image layer where the application is built and packaged, the Kubernetes workload layer where Deployments, StatefulSets, and autoscaling run it, and the external access layer where Ingress and TLS expose it to the world. Just as ticking the final checklist confirms match-readiness, completing the lab checklist confirms every layer actually works together. The payoff: you can explain the entire platform end to end — the mark of real understanding, and the foundation M5's security work builds on.

bash

# Final checklist and security architecture summary

echo "=== M5 Lab — Final Verification Checklist ==="

PASS=0; FAIL=0
check() {
  local desc=$1; local cmd=$2; local expected=$3
  result=$(eval "$cmd" 2>&1)
  if echo "$result" | grep -q "$expected"; then
    echo "PASS ✓ $desc"; ((PASS++))
  else
    echo "FAIL ✗ $desc"; ((FAIL++))
  fi
}

check "Cosign policy in Enforce mode"   "kubectl get clusterpolicy require-cosign-signature -o jsonpath='{.spec.validationFailureAction}'"   "Enforce"

check "Seccomp policy active"   "kubectl get clusterpolicy require-runtime-default-seccomp -o jsonpath='{.spec.validationFailureAction}'"   "Enforce"

check "Ingress TLS policy active"   "kubectl get clusterpolicy require-ingress-tls -o jsonpath='{.spec.validationFailureAction}'"   "Enforce"

check "PSA restricted on production"   "kubectl get ns production -o jsonpath='{.metadata.labels.pod-security\.kubernetes\.io/enforce}'"   "restricted"

check "Default-deny NetworkPolicy exists"   "kubectl get networkpolicy default-deny-all -n production -o name"   "networkpolicy.networking.k8s.io/default-deny-all"

check "API serving correctly"   "kubectl exec -n production $(kubectl get pod -l app=ipl-api -n production -o name | head -1) -- python3 -c 'import urllib.request; r=urllib.request.urlopen("http://localhost:8000/health"); print(r.status)'"   "200"

echo ""
echo "Results: $PASS PASS, $FAIL FAIL"
echo ""
echo "=== Security Architecture: Threat Coverage Map ==="
echo ""
echo "RBAC + ServiceAccounts     → API server compromise, over-privileged workloads"
echo "Pod Security Admission     → Container escape, privilege escalation"
echo "NetworkPolicies            → Lateral movement, east-west traffic abuse"
echo "Kyverno supply chain       → Supply chain attacks, tampered images"
echo "Kyverno security policies  → Missing hardening settings, seccomp bypass"
echo "Kyverno governance         → Operational drift, unapproved storage/ingress"
echo "Vault dynamic secrets      → Credential leak blast radius, static key exposure"
echo "IRSA/Workload Identity     → Long-lived cloud credential storage"
echo ""
echo "Course 3 Module 5: Kubernetes Security complete ✓"

Warning: Setting `background: false` on a Kyverno policy with `verifyImages` (like the supply chain policy) means the policy only evaluates Pods at admission time — existing running Pods that were deployed before the policy was installed are not evaluated. This is intentional for supply chain policies because re-evaluating existing Pods against a new signature requirement would not revoke their execution anyway. However, it means the `background: false` policy provides no protection against Pods that were deployed during a Kyverno outage or before the policy was first installed. Combine the admission-time policy with a scheduled `kubectl kyverno apply` scan in CI to detect any non-compliant Pods that slipped through, running the policy evaluation as an audit scan against the live cluster state rather than only at admission time.

Extension Challenge: Implement a Kyverno `validate` policy that checks the `kube-bench` CIS Kubernetes Benchmark score for every new Node joining the cluster using the Node admission webhook, requiring a minimum CIS Level 1 compliance score before the Node is permitted to receive workloads. This requires writing a Kyverno policy that calls an external endpoint (the kube-bench score service) using Kyverno's `external data` feature, implementing the most advanced Kyverno capability: context-enriched policies that consult external data sources during admission evaluation. The external data pattern is the bridge between Kyverno's rule engine and any organisational data store, opening a vast space of context-aware governance policies that static YAML patterns cannot express.

Policy-as-code enforcement shifts security controls from detective (post-deployment audits) to preventive (admission-time rejection), making it structurally impossible to deploy non-compliant resources through normal GitOps workflows.
The four policy domains — supply chain (Cosign), security hardening (seccomp, non-root), operational governance (Ingress TLS, StorageClass), and automated remediation (labels, quota generation) — each address distinct governance categories that no single existing control covers.
Set `background: false` on supply chain signature verification policies to avoid re-evaluating existing resources; use scheduled `kubectl kyverno apply` scans to audit existing resources periodically.
The complete M5 security stack — RBAC, PSA, NetworkPolicies, Kyverno — implements defence in depth: each layer independently limits the blast radius of a compromise, so bypassing one layer does not eliminate all protection.
All Kyverno policies should be tested locally with `kubectl kyverno apply` against both compliant and non-compliant manifests before cluster deployment, and deployed in Audit mode with PolicyReport review before switching to Enforce.
The security architecture threat coverage map — mapping each control to the threat category it addresses — is the communication artefact that makes the security model legible to stakeholders, auditors, and new team members.