Lab — block unsigned images from deploying to EKS via Kyverno — CI/CD, GitOps, DevSecOps & Observability

What You'll Build

In this lab you will install Kyverno on your EKS cluster, manage it via the App of Apps GitOps pattern, and deploy a ClusterPolicy that requires Cosign signatures on all india-squad images admitted to the squad-prod namespace. You will test the policy in Audit mode to observe violations without blocking, promote it to Enforce mode, and verify that an unsigned image is rejected while a signed image from Exercise 20's pipeline is admitted.

You will also verify that the Kyverno policy is itself managed by GitOps: manually downgrading the policy to Audit mode via kubectl and watching ArgoCD's self-heal revert it back to Enforce within the sync interval, demonstrating that even admission control policies are protected against manual cluster overrides by the GitOps workflow, confirming that the GitOps workflow governs security policy as strictly as application code.

Analogy🏏Cricket

Think of it like cricket: Picture setting up a remote ground management system for a cricket ground in a new city. First, the infrastructure must be installed: the pitch sensors, the scoreboard network, and the broadcast uplink—equivalent to installing ArgoCD on the EKS cluster. Then, the venue configuration must be committed to the central venue management database: the pitch dimensions, the lighting schedule, the boundary positions—equivalent to committing the Kubernetes manifests to the GitOps repository. Then, the venue must be registered with the central management system, which then automatically enforces the declared configuration at the ground—equivalent to creating the ArgoCD Application that connects the repository to the cluster. When a ground manager moves a boundary rope by hand, the sensors detect the drift and alert the management system to restore the declared position—equivalent to ArgoCD detecting and reverting the manual replica scale. This reveals why the lab sequence matters: you cannot verify GitOps until all three components—operator, repository, and Application—are connected and working together.

Prerequisites

EKS cluster with ArgoCD from Module 2 Lab 14, with the App of Apps structure in the india-squad-gitops repository.
The squad-prod namespace created in the cluster from the same lab.
A signed container image from Exercise 20's DevSecOps pipeline—the image must have been signed by the MY_ORG/india-squad-helm build.yml workflow using Cosign.
Helm 3 and the argocd CLI installed locally for the Kyverno installation and App of Apps GitOps integration steps.
Cosign CLI installed locally for the final verification step that confirms the signature is valid outside of Kyverno's admission check.

Setup — Install Kyverno via Helm and GitOps

Install Kyverno directly with Helm first to get it running, then add it to the App of Apps GitOps structure so subsequent updates are managed via Git. The three-replica admission controller configuration ensures that Kyverno's webhook remains available during rolling updates and node failures—a single-replica Kyverno with `failurePolicy: Fail` on the validating webhook would cause cluster-wide admission failures if the Kyverno pod restarts.

Analogy🏏Cricket

🏏 Think of it like cricket: You post gate stewards for a big match by first getting them physically in place, then writing them into the official duty roster so every future shift is scheduled from the roster — exactly as you install Kyverno with Helm to get it running, then add it to the App of Apps so subsequent updates flow through Git. Just as you always roster at least three stewards on the gate so the entry check never fails during a shift change, you run three Kyverno admission-controller replicas so the webhook stays available through rolling updates and node failures. Just as a single steward told 'let no one in if unsure' would lock the whole crowd out the moment they step away, a single-replica Kyverno with failurePolicy Fail would block every admission cluster-wide if it went down. The payoff: redundant, Git-managed enforcement stays highly available without ever becoming a single point of failure.

bash

# Prerequisites: EKS cluster with ArgoCD installed (from M2 Lab 14).
# We will install Kyverno and deploy admission policies via GitOps.

# 1. Install Kyverno on the EKS cluster using Helm
helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

helm upgrade --install kyverno kyverno/kyverno \
  --namespace kyverno --create-namespace \
  --set admissionController.replicas=3 \
  --set backgroundController.replicas=2 \
  --set reportsController.replicas=2 \
  --wait

# Verify Kyverno is running
kubectl get pods -n kyverno
# Expected: kyverno-admission-controller-*, kyverno-background-controller-*
#           kyverno-cleanup-controller-*, kyverno-reports-controller-*

# 2. Add Kyverno to the GitOps repository as an App of Apps child
# (using the ArgoCD structure from M2 Lab 14)
mkdir -p india-squad-gitops/apps
cat > india-squad-gitops/apps/kyverno.yaml << 'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kyverno
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://kyverno.github.io/kyverno/
    chart: kyverno
    targetRevision: '3.x.x'
    helm:
      values: |
        admissionController:
          replicas: 3
  destination:
    server: https://kubernetes.default.svc
    namespace: kyverno
  syncPolicy:
    automated: { prune: true, selfHeal: true }
    syncOptions: [CreateNamespace=true]
EOF
cd india-squad-gitops && git add apps/kyverno.yaml
git commit -m 'feat: add Kyverno to App of Apps'
git push

Step 1 — Deploy Image Signing Policy via GitOps

Add the ClusterPolicy to the GitOps repository in Audit mode and create an ArgoCD Application that manages the policies directory. Starting in Audit mode is critical: it reveals how many existing pods in squad-prod are running unsigned images before the policy blocks new admissions, giving the team time to ensure the signed image pipeline is working before enforcement begins.

Analogy🏏Cricket

Think of it like cricket: Deploying the policy in Audit mode is the equivalent of the ICC's grace period when introducing a new playing condition. During the grace period, violations are recorded and reported to team managers but the match is not stopped. Managers can see exactly how often and where the violation occurs and prepare their players accordingly. Only after the grace period does the umpire enforce the condition in real time. Just as the grace period prevents a new condition from disrupting a match before teams have adapted, Audit mode prevents the signing policy from blocking deployments before the image signing pipeline is confirmed working.

bash

# Step 1: Create and deploy the image signing ClusterPolicy via GitOps.

# Add the policy to the GitOps repository
mkdir -p india-squad-gitops/policies
cat > india-squad-gitops/policies/require-image-signature.yaml << 'EOF'
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-india-squad-signature
  annotations:
    policies.kyverno.io/title: Require India Squad Image Signature
    policies.kyverno.io/category: Supply Chain Security
spec:
  validationFailureAction: Audit   # start in Audit mode
  background: true
  rules:
    - name: verify-cosign-signature
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [squad-prod]
      verifyImages:
        - imageReferences:
            - '123456789.dkr.ecr.ap-south-1.amazonaws.com/india-squad-api*'
          attestors:
            - entries:
                - keyless:
                    subject: >-
                      https://github.com/MY_ORG/india-squad-helm/.github/workflows/devsecops.yml@refs/heads/main
                    issuer: 'https://token.actions.githubusercontent.com'
EOF

# Create an ArgoCD Application for the policies directory
cat > india-squad-gitops/apps/squad-policies.yaml << 'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: squad-policies
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/MY_ORG/india-squad-gitops
    targetRevision: main
    path: policies/
  destination:
    server: https://kubernetes.default.svc
    namespace: kyverno
  syncPolicy:
    automated: { prune: true, selfHeal: true }
EOF

cd india-squad-gitops
git add policies/ apps/squad-policies.yaml
git commit -m 'feat: add image signing policy in Audit mode'
git push

# Verify the policy was applied by ArgoCD
argocd app sync squad-policies
kubectl get clusterpolicy require-india-squad-signature

Step 2 — Test Audit Mode, Then Enforce

Deploy an unsigned nginx image to confirm it is admitted in Audit mode and that the violation is recorded in the PolicyReport. Check the PolicyReport resource to see the recorded violation. Then promote the policy to Enforce mode by updating the GitOps repository and pushing the change, which ArgoCD applies automatically. Verify the policy mode changed by querying the ClusterPolicy status.

Analogy🏏Cricket

Think of it like cricket: Checking the PolicyReport for the unsigned pod's violation is like reviewing the trial season's observations before the ICC enforces a new condition. The observations reveal: 'five deliveries would have been called no-balls under the new condition in three matches involving two different bowlers.' The selectors can now brief those bowlers before enforcement begins. Promoting to Enforce via a Git commit is the equivalent of the ICC publishing the formal notification that the condition is now enforced internationally from the next series. The change is documented, reviewable, and the effective date is recorded in the Git history.

bash

# Step 2: Test Audit mode — deploy unsigned image and check the PolicyReport.

# Deploy an unsigned nginx image to squad-prod (should PASS in Audit mode)
kubectl apply -f - << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: unsigned-test
  namespace: squad-prod
spec:
  containers:
    - name: nginx
      image: nginx:latest
  restartPolicy: Never
EOF

# The pod is created (Audit mode does not block)
kubectl get pod unsigned-test -n squad-prod

# Check the PolicyReport for the violation record
kubectl get policyreport -n squad-prod
kubectl describe policyreport -n squad-prod
# Expected: result: fail for unsigned-test / verify-cosign-signature

# Clean up the test pod
kubectl delete pod unsigned-test -n squad-prod

# Now promote the policy to Enforce mode
sed -i 's/validationFailureAction: Audit/validationFailureAction: Enforce/' \
  india-squad-gitops/policies/require-image-signature.yaml
cd india-squad-gitops
git add policies/require-image-signature.yaml
git commit -m 'security: promote image signing policy to Enforce mode'
git push
argocd app sync squad-policies
kubectl get clusterpolicy require-india-squad-signature -o jsonpath='{.spec.validationFailureAction}'
# Expected: Enforce

Step 3 — Verify Blocking and Admission

Verify the two outcomes that confirm the policy is working correctly: an unsigned image is blocked with a clear error message, and the signed image from Exercise 20 is admitted and its tag is mutated to a digest by Kyverno's `mutateDigest: true` option. The digest mutation is the supply chain security guarantee: even if someone replaces the image behind the tag after signing, the pod spec is permanently bound to the signed bytes.

Analogy🏏Cricket

Think of it like cricket: The unsigned image rejection is the umpire calling a no-ball: the delivery is illegal, play stops, and the team is told clearly why. The signed image admission is the umpire confirming the delivery is legal: play continues. The digest mutation is the match official recording the exact ball specification in the match log after the delivery is bowled: the record is immutable. Just as the match log's ball specification prevents anyone from later claiming the delivery used a different ball, the pod spec's digest reference prevents the container runtime from later running a different image if the tag is overwritten. This reveals why mutateDigest is not optional in a supply chain security policy: without it, the signing verification is a point-in-time check, not a persistent guarantee.

bash

# Step 3: Verify Enforce mode blocks unsigned images.

# Attempt to deploy the unsigned nginx image (should be REJECTED)
kubectl apply -f - << 'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: unsigned-test
  namespace: squad-prod
spec:
  containers:
    - name: nginx
      image: nginx:latest
  restartPolicy: Never
EOF
# Expected error:
# Error from server: admission webhook 'mutate.kyverno.svc-fail' denied the request:
# resource Pod/squad-prod/unsigned-test was blocked due to the following policies:
# require-india-squad-signature:
#   verify-cosign-signature: image nginx:latest not found in rules

# Now deploy the SIGNED image from the DevSecOps pipeline (should PASS)
# Replace <sha> with the actual commit SHA from the Exercise 20 pipeline
SIGNED_IMAGE='123456789.dkr.ecr.ap-south-1.amazonaws.com/india-squad-api:<sha>'

kubectl apply -f - << EOF
apiVersion: v1
kind: Pod
metadata:
  name: signed-test
  namespace: squad-prod
spec:
  containers:
    - name: api
      image: $SIGNED_IMAGE
  restartPolicy: Never
EOF
# Expected: pod/signed-test created
kubectl get pod signed-test -n squad-prod
# Note the image in the pod spec — Kyverno mutateDigest replaces the tag with digest
kubectl get pod signed-test -n squad-prod \
  -o jsonpath='{.spec.containers[0].image}'
# Expected: .../india-squad-api@sha256:...  (digest, not tag)

Step 4 — End-to-End Verification

Run the complete verification suite: confirm the policy mode, confirm unsigned images are rejected, confirm the signed image's digest reference, verify the Cosign signature from the CLI, and perform the GitOps self-heal test by manually downgrading the policy and watching ArgoCD restore the Enforce mode. The self-heal test is the final proof that the supply chain enforcement is itself protected by the GitOps workflow.

Analogy🏏Cricket

🏏 Think of it like cricket: You don't declare a new umpiring system fit for a final until you've checked every path end to end — the scoreboard matches the official record, the fielders match the declared plan, and, most important, a review can actually overturn a wrong decision. That is exactly the ArgoCD verification: confirm the Application is Synced and Healthy, the running image matches the Git-declared tag, the replica count matches the production overlay, and a git revert genuinely rolls the deployment back. Just as testing the review overturn matters most because it proves you can undo a mistake under pressure, the git revert test matters most because it proves the rollback path works. Just as a verified system lets the match proceed with confidence, a verified GitOps loop lets the team ship knowing every change is reversible. The payoff: proving the revert works is what makes the whole pipeline trustworthy.

bash

# Step 4: Verify the complete supply chain enforcement chain.

# 1. Confirm the ClusterPolicy is Enforce and healthy
kubectl get clusterpolicy require-india-squad-signature \
  -o jsonpath='{.spec.validationFailureAction}'
# Expected: Enforce

kubectl get clusterpolicy require-india-squad-signature \
  -o jsonpath='{.status.conditions[0].status}'
# Expected: True (policy is ready)

# 2. Confirm unsigned pod is rejected
kubectl run unsigned-verify --image=python:3.12 -n squad-prod --restart=Never 2>&1 | grep 'blocked'
# Expected: 'was blocked due to the following policies'

# 3. Confirm signed pod is admitted and image is pinned to digest
kubectl get pod signed-test -n squad-prod \
  -o jsonpath='{.spec.containers[0].image}' | grep 'sha256'
# Expected: contains '@sha256:' — Kyverno mutated the tag to digest

# 4. Verify the Cosign signature from the CLI
IMAGE=$(kubectl get pod signed-test -n squad-prod \
  -o jsonpath='{.spec.containers[0].image}')
cosign verify \
  --certificate-identity-regexp='https://github.com/MY_ORG/india-squad-helm' \
  --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
  $IMAGE
# Expected: Verification for ...: true

# 5. Confirm GitOps manages the policy (rollback test)
# Change Enforce back to Audit directly in kubectl:
kubectl patch clusterpolicy require-india-squad-signature \
  --type=json -p='[{"op":"replace","path":"/spec/validationFailureAction","value":"Audit"}]'
kubectl get clusterpolicy require-india-squad-signature \
  -o jsonpath='{.spec.validationFailureAction}'
# Expected: Audit (manual change applied)

# ArgoCD self-heal reverts it back to Enforce within the sync interval:
sleep 180   # wait for ArgoCD sync
kubectl get clusterpolicy require-india-squad-signature \
  -o jsonpath='{.spec.validationFailureAction}'
# Expected: Enforce (ArgoCD reverted the manual change)

Warning: Configure a namespace exclusion for the kyverno namespace itself in the verifyImages policy. If Kyverno's own pods need to restart and the verifyImages policy applies to all namespaces including kyverno, the Kyverno pods may fail to restart because their images are not signed with your organisation's Cosign identity. This creates a deadlock where Kyverno is unavailable but also preventing itself from restarting. Add `namespaces: [squad-prod, squad-staging]` to the match block and explicitly exclude system namespaces from all image verification policies.

Extension Challenge: Add a Kyverno PSS Baseline policy to the squad-prod namespace that prevents privileged containers, hostPath volumes, and host network access. Test it by attempting to deploy a pod with `securityContext.privileged: true` and confirming it is rejected. Then add a Kyverno mutating policy for the squad-staging namespace that automatically adds `runAsNonRoot: true` and `allowPrivilegeEscalation: false` to containers that omit these fields, and verify the mutation is applied by inspecting the pod spec after admission.