Policy Practice — Enforce a Cluster Policy — DevSecOps & Site Reliability Engineering

What You'll Build

In this exercise you will install Kyverno on a local or staging Kubernetes cluster, deploy three ClusterPolicies covering image registry allowlisting, non-root enforcement, and resource limits, test them in Audit mode against the CricketPulse deployment, remediate all violations, and switch to Enforce mode. You will then add a conftest OPA check to the CI pipeline so violations are caught at PR time, before they reach the cluster.

Analogy🏏Cricket

🏏 Think of it like cricket: This is a full-match simulation in the nets — a structured practice match with umpires, scorers, and a target, played under match conditions so the team builds muscle memory for the real thing. The coach throws curveballs to test the team's response to unexpected events. After the simulation, the team watches the footage and identifies improvements for the next real match.

Prerequisites

Lessons 9, 10, and 11 completed.
A Kubernetes cluster (minikube, kind, or a staging cluster).
kubectl configured to the target cluster.
Helm 3 installed.
conftest installed (brew install conftest or from GitHub releases).
The CricketPulse deployment YAML from Module 2 exercises.

Step 1 — Install Kyverno

Install Kyverno using Helm with a 3-replica configuration for high availability. Verify all Kyverno pods are running before proceeding to policy deployment.

Analogy🏏Cricket

🏏 Think of it like cricket: Before the match officials can enforce the Laws, they need to be physically present on the ground. Installing Kyverno is like positioning the on-field umpires and third umpire before the first ball is bowled.

bash

# Install Kyverno
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno   --namespace kyverno   --create-namespace   --set admissionController.replicas=3   --set backgroundController.replicas=2

# Wait for Kyverno to be ready
kubectl rollout status deployment/kyverno-admission-controller -n kyverno
kubectl rollout status deployment/kyverno-background-controller -n kyverno

# Verify webhook configuration is registered
kubectl get validatingwebhookconfigurations | grep kyverno
kubectl get mutatingwebhookconfigurations | grep kyverno

# Expected output:
# kyverno-policy-validating-webhook-cfg    ...
# kyverno-resource-validating-webhook-cfg  ...
# kyverno-policy-mutating-webhook-cfg      ...

Step 2 — Deploy Policies in Audit Mode

Deploy the three ClusterPolicies with validationFailureAction: Audit. In Audit mode, violations are recorded in PolicyReports but do not block deployments. This lets you assess the impact of the policies against your existing workloads before enforcement.

Analogy🏏Cricket

🏏 Think of it like cricket: The umpires observe a practice match using the new Laws but don't call any dismissals — they record what would have been called under the new Laws so the players can see how their current technique performs before the rules are enforced in a real match.

yaml

# File: kyverno-policies/registry-allowlist.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  validationFailureAction: Audit   # change to Enforce after remediation
  rules:
    - name: check-registry
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        message: "Images must come from ghcr.io/myorg/ or 123456.dkr.ecr.ap-south-1.amazonaws.com/"
        pattern:
          spec:
            containers:
              - image: "ghcr.io/myorg/* | 123456.dkr.ecr.ap-south-1.amazonaws.com/*"

---
# File: kyverno-policies/require-nonroot.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-nonroot
spec:
  validationFailureAction: Audit
  rules:
    - name: check-runAsNonRoot
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        message: "All containers must set securityContext.runAsNonRoot: true"
        pattern:
          spec:
            containers:
              - securityContext:
                  runAsNonRoot: true

---
# File: kyverno-policies/require-resource-limits.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Audit
  rules:
    - name: check-cpu-limit
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        message: "All containers must specify resources.limits.cpu and resources.limits.memory"
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    cpu: "?*"
                    memory: "?*"

bash

# Apply all policies
kubectl apply -f kyverno-policies/

# Deploy the CricketPulse workload (will not be blocked in Audit mode)
kubectl apply -f k8s/ -n cricketpulse-staging

# Wait and check the PolicyReport
sleep 30
kubectl get policyreport -n cricketpulse-staging
kubectl describe policyreport -n cricketpulse-staging

# Sample PolicyReport output:
# Policy                  Result  Category  Message
# restrict-image-registries FAIL  Images    Image nginx:latest not in allowlist
# require-nonroot           FAIL  Security  Container 'api' does not set runAsNonRoot
# require-resource-limits   PASS  Resources All containers have limits

Step 3 — Remediate Violations and Switch to Enforce

Fix each violation identified in the PolicyReport, verify a clean report, then switch all policies to Enforce mode. Test that the Enforce mode correctly blocks a non-compliant deployment.

Analogy🏏Cricket

🏏 Think of it like cricket: The batting coach reviews the list of technical violations from the practice session and works with each batter to correct them. Once all flaws are fixed in training, the same Laws are enforced in the real match with full consequences for violations.

yaml

# Fix 1: Update Kubernetes deployment to use approved registry image
# k8s/deployment.yaml — change image
containers:
  - name: api
    image: ghcr.io/myorg/cricketpulse:latest  # was: nginx:latest

# Fix 2: Add security context to all containers
containers:
  - name: api
    image: ghcr.io/myorg/cricketpulse:latest
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: [ALL]
    resources:
      limits:
        cpu: "500m"
        memory: "512Mi"
      requests:
        cpu: "100m"
        memory: "128Mi"

# Redeploy and verify clean PolicyReport
kubectl apply -f k8s/ -n cricketpulse-staging
sleep 30
kubectl get policyreport -n cricketpulse-staging -o json   | python3 -c "
import json,sys
r = json.load(sys.stdin)
for item in r.get('items',[]):
    results = item.get('results',[])
    fails = [r for r in results if r['result']=='fail']
    print(f'Failures: {len(fails)}')
"
# Expected: Failures: 0

bash

# Switch all policies to Enforce mode
kubectl patch clusterpolicy restrict-image-registries   --type merge -p '{"spec":{"validationFailureAction":"Enforce"}}'
kubectl patch clusterpolicy require-nonroot   --type merge -p '{"spec":{"validationFailureAction":"Enforce"}}'
kubectl patch clusterpolicy require-resource-limits   --type merge -p '{"spec":{"validationFailureAction":"Enforce"}}'

# Test: attempt to deploy non-compliant pod (should be blocked)
kubectl run policy-test   --image=nginx:latest   --restart=Never   -n cricketpulse-staging
# Expected: Error from server: admission webhook denied the request:
#   Images must come from ghcr.io/myorg/ or 123456.dkr.ecr.ap-south-1.amazonaws.com/

Step 4 — Add conftest to CI Pipeline

Add a conftest step to the CI pipeline that evaluates Kubernetes manifests against OPA policies before they reach the cluster. This catches violations at PR review time — minutes before the deployment, not after it.

Analogy🏏Cricket

🏏 Think of it like cricket: adding conftest to CI is like moving the equipment check from the boundary edge to the dressing room before the player ever walks out. Just as an official inspecting bats and pads at the pavilion gate catches an illegal bat minutes before the batsman faces a ball — rather than calling him back after he's already scored — a conftest step evaluates Kubernetes manifests against OPA policies before they reach the cluster, catching violations at PR review time instead of after deployment. Just as stopping the illegal kit early saves an awkward mid-innings recall and a disputed scorecard, catching the violation in CI saves a risky rollback in production. The payoff: bad configuration is turned away at the gate, so only compliant workloads make it onto the field.

rego

# Write conftest/OPA policies equivalent to the Kyverno policies
# File: policies/k8s/security.rego
package kubernetes.security

deny[msg] {
    input.kind == "Deployment"
    container := input.spec.template.spec.containers[_]
    not startswith(container.image, "ghcr.io/myorg/")
    not startswith(container.image, "123456.dkr.ecr.ap-south-1.amazonaws.com/")
    msg := sprintf("Container '%v' uses unapproved registry: %v", [container.name, container.image])
}

deny[msg] {
    input.kind == "Deployment"
    container := input.spec.template.spec.containers[_]
    not container.securityContext.runAsNonRoot == true
    msg := sprintf("Container '%v' must set runAsNonRoot: true", [container.name])
}

deny[msg] {
    input.kind == "Deployment"
    container := input.spec.template.spec.containers[_]
    not container.resources.limits.cpu
    msg := sprintf("Container '%v' must set resources.limits.cpu", [container.name])
}

# GitHub Actions CI step
- name: Conftest policy check
  run: |
    conftest test k8s/ --policy policies/k8s/ --output table
    if [ $? -ne 0 ]; then
      echo "FAIL: Kubernetes manifests violate OPA policies"
      exit 1
    fi

Verify Your Work

Run the complete verification checklist to confirm all three policies are active in Enforce mode and conftest catches violations in CI.

Analogy🏏Cricket

🏏 Think of it like cricket: A practice session is worthless until you grade it against clear benchmarks. Just as a batting coach scores a net session on measurable targets — did the batter rotate strike, was the trigger movement quick enough, did they leave the balls outside off — you review the drill against five concrete criteria rather than a vague sense that 'it went okay'. Just as the coach times how fast the batter read the length (time to detect, target under ten minutes) and how quickly they adjusted their shot (time to contain, target under twenty), you measure detection and containment against fixed thresholds. Just as a session only passes if every benchmark is met, not most of them, a passing drill must satisfy all five: fast detection, fast containment, status page updated, a systemic root cause, and at least three owned action items. The payoff: honest scoring against a rubric shows exactly which reflex to sharpen before the real match, rather than leaving you falsely confident.

bash

# Full verification checklist

# 1. Kyverno webhook is healthy
kubectl get validatingwebhookconfigurations kyverno-resource-validating-webhook-cfg   -o jsonpath='{.webhooks[0].failurePolicy}'
# Expected: Fail

# 2. All policies in Enforce mode
for policy in restrict-image-registries require-nonroot require-resource-limits; do
  mode=$(kubectl get clusterpolicy $policy -o jsonpath='{.spec.validationFailureAction}')
  echo "$policy: $mode"
done
# Expected: all show Enforce

# 3. Non-compliant pod is blocked
kubectl run test-block --image=docker.io/nginx --restart=Never -n cricketpulse-staging 2>&1
# Expected: Error from server

# 4. Compliant pod is allowed
kubectl run test-allow   --image=ghcr.io/myorg/cricketpulse:latest   --restart=Never   --overrides='{"spec":{"containers":[{"name":"test-allow","image":"ghcr.io/myorg/cricketpulse:latest","securityContext":{"runAsNonRoot":true,"runAsUser":1000},"resources":{"limits":{"cpu":"100m","memory":"128Mi"}}}]}}'   -n cricketpulse-staging
# Expected: pod/test-allow created

# 5. conftest catches violation before cluster
echo 'apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-violation
spec:
  template:
    spec:
      containers:
        - name: api
          image: nginx:latest' > /tmp/violation.yaml
conftest test /tmp/violation.yaml --policy policies/k8s/
# Expected: FAIL - 2 violations found

Before switching to Enforce mode, also check DaemonSets, StatefulSets, Jobs, and CronJobs — not just Deployments. Kyverno policies for Pod kind match the pod spec template inside all workload types. A DaemonSet pod template that violates the policy will block the DaemonSet from rolling out, which can affect node-level services like log collectors and monitoring agents.

Kyverno's PolicyReport CRD aggregates all policy results per namespace and updates continuously. You can set up a Grafana dashboard over the Kyverno metrics endpoint (port 8000 on Kyverno pods) to visualise policy pass/fail ratios over time — a useful compliance KPI dashboard showing your cluster's policy compliance trend.

Install Kyverno with 3 replicas for high availability — a single-replica admission controller is a single point of failure.
Deploy policies in Audit mode first, collect PolicyReport violations, remediate, then switch to Enforce.
Add conftest to CI to catch manifest violations at PR time — before the manifest reaches the cluster.
Check all workload types (Deployment, DaemonSet, StatefulSet, Job) when auditing for policy violations.
Set failurePolicy: Fail on the Kyverno webhook and exclude kyverno and kube-system namespaces.