Lab — SBOM pipeline, in-toto attestation and Kyverno admission enforcement

What You'll Build

In this lab you will implement the complete container supply chain security pipeline that transforms a passing CI build into a cryptographically attested, registry-signed, admission-webhook-enforced deployment artefact. The pipeline extends the GitHub Actions workflow from M1 Lesson 7 with four new supply chain steps: SBOM generation using Syft in CycloneDX format, SBOM signing and attachment to the image digest as an OCI referrer using Cosign, SLSA provenance attestation using the slsa-github-generator reusable workflow, and Kyverno ClusterPolicy deployment that enforces signature verification at pod admission time. When complete, any attempt to deploy an unsigned or improperly attested image to the target Kubernetes cluster will be rejected by the admission webhook with a descriptive policy violation message, completing the supply chain integrity chain from source commit to running pod.

The structure of this lab is designed so that each step adds a new, independently testable supply chain property: the SBOM step adds software composition transparency; the Cosign signing step adds identity-bound tamper evidence; the SLSA provenance step adds build-process verifiability; and the Kyverno enforcement step converts all three from documentation to enforcement. Each step can be verified independently before the next is added, which is the correct incremental rollout strategy for supply chain controls in a live environment — replacing all four simultaneously with a single Kyverno Enforce policy would immediately break all deployments that had not yet been signed, while the incremental approach allows each control to be validated before its enforcement is activated.

Analogy🏏Cricket

🏏 Think of it like cricket: This lab's staging-then-production TLS pipeline is the ICC's pre-tour practice match protocol. Before the Test series proper, the touring team plays a two-day warm-up match against a local state side — not under ICC playing conditions, not with the official match balls, and not counted in official records. The warm-up match validates that the team's batting and bowling combinations work on local pitch conditions before committing to the conditions for the official five-day Test. The Let's Encrypt staging environment is that warm-up match: it issues real certificates signed by a staging CA (not trusted by browsers, like an unofficial match result), validates the complete ACME challenge flow, and confirms that DNS, network, and IAM configurations are correct — all without consuming the production rate limit. Switching to the production issuer is committing to the official Test: the certificate is now signed by Let's Encrypt's trusted CA (officially recognised), but any misconfiguration wastes an official certificate issuance. The certificate rotation simulation is the team testing their emergency substitution protocol — specifically waiting until the 'player' (certificate) is within the renewal window, confirming the automatic replacement process fires correctly, and verifying the new 'player' is ready to take the field. This reveals why the three-phase structure of the lab matters: each phase validates a distinct property of the TLS chain, and proceeding through them in order reduces the risk that a configuration problem discovered in production was one that staging testing would have caught.

Prerequisites

The working GitHub Actions pipeline from M1 Lesson 7 that builds, scans with Trivy, and pushes to ECR — this lab adds four new jobs to that workflow rather than replacing it.
A Kubernetes cluster with Kyverno installed — `kubectl apply -f https://github.com/kyverno/kyverno/releases/latest/download/install.yaml` installs Kyverno in the `kyverno` namespace with default configuration.
Cosign v2 installed and `cosign tree` verified working against the ECR repository from M1 — confirms the ECR registry supports OCI referrers API used for signature and attestation storage.
Syft v0.100+ installed locally — `brew install syft` on macOS; used to verify SBOM generation locally before the CI step is added.
The `slsa-github-generator` reusable workflow accessible from your repository — requires the GitHub Actions OIDC permission and a GitHub App token with workflow permissions for cross-repository workflow calls.

Setup & Project Structure

Verify the baseline pipeline from M1 Lesson 7 produces a clean Trivy scan and a successfully pushed image before adding any supply chain steps. Adding SBOM generation and signing to a pipeline that has existing failures creates compound debugging problems where the new failure may mask the existing one. The correct engineering discipline is: establish a clean baseline, verify it, then add each new step one at a time, verifying again after each addition.

Analogy🏏Cricket

🏏 Think of it like cricket: setting up a tournament venue is not one job but a strict sequence of jobs, and smart boards hire a professional event crew instead of doing each task by hand. Just as the event crew handles seating, accreditation desks, and broadcast rigging as one coordinated package, Helm installs the Nginx IngressController and cert-manager with all their CRDs, ServiceAccounts, and RBAC in one release instead of dozens of hand-applied manifests. And order matters: just as the DRS cameras and replay screens must be rigged and tested before the third umpire takes his seat — an official with no working replay feed is useless and the review system collapses — cert-manager's CRDs must exist before the controller starts, or it crashes on startup before its CRD admission webhooks ever come online. The payoff: a correctly sequenced, fully provisioned venue — controller infrastructure that comes up cleanly the first time.

bash

# Verify baseline pipeline before extending with supply chain steps

# ── 1. Confirm ECR image exists from M1 Lab ───────────────────────────────
ECR_REGISTRY="123456789.dkr.ecr.ap-south-1.amazonaws.com"
ECR_REPO="ipl-scorecard"

aws ecr describe-images   --repository-name $ECR_REPO   --region ap-south-1   --query 'imageDetails[?contains(imageTags, `latest`)].[imageDigest,imagePushedAt]'   --output table
# Must show at least one image tagged 'latest' before proceeding

# ── 2. Verify Cosign can query ECR referrers API ──────────────────────────
IMAGE="$ECR_REGISTRY/$ECR_REPO:latest"
cosign tree $IMAGE
# If this returns "WARN: no signatures associated" rather than an auth error,
# the baseline is correct — no signatures yet, but ECR supports the referrers API

# ── 3. Generate SBOM locally to test Syft before CI integration ──────────
syft $IMAGE   --output cyclonedx-json=local-test-sbom.json
# Must produce a valid CycloneDX JSON — check component count
python3 -c "
import json
with open('local-test-sbom.json') as f: sbom = json.load(f)
print(f'SBOM components: {len(sbom.get("components", []))}')
# Expected: 50-200 components depending on python:3.11-slim base
"

# ── 4. Install Kyverno in the test cluster (kind or minikube) ─────────────
kubectl apply -f https://github.com/kyverno/kyverno/releases/latest/download/install.yaml
kubectl wait --for=condition=ready pod   -l app.kubernetes.io/name=kyverno   -n kyverno   --timeout=120s
echo "Kyverno ready ✓"

# ── 5. Deploy Kyverno policy in Audit mode first ──────────────────────────
# (defined in Step 3 below — deploy here to capture Audit violations during lab)
echo "Baseline verified — ready to add supply chain steps"

Step 1 — Foundation

Add the SBOM generation step to the GitHub Actions workflow immediately after the Docker push step, using Syft to generate a CycloneDX SBOM and Cosign to attach it to the image as an OCI artifact. The SBOM attachment stores the SBOM in the same ECR repository as the image, addressable through the OCI referrers API at the image's digest, without any additional storage configuration. Setting up SBOM generation before signature generation is deliberate — the SBOM is input to the vulnerability attestation in a later step, and generating it first confirms that ECR's referrers API is accessible from the CI runner before adding the more complex signing ceremony.

Analogy🏏Cricket

🏏 Think of it like cricket: before the first official fixture, a serious team plays a full-dress practice match — real pitch, real umpires, real match conditions — knowing the result won't appear in any league table. Just as that practice match earns no points but proves the batting order, bowling plans, and fielding drills actually work under match pressure, the Let's Encrypt staging certificate won't be trusted by any browser but proves that the ACME challenge completes, DNS resolves correctly, and the IngressController routes traffic as designed. Just as a coach would never debut an untested game plan in a televised final, you never point at the production issuer first — a failed live attempt costs far more than a failed rehearsal. The payoff: every moving part of the certificate pipeline validated cheaply, so the later switch to production is a formality rather than a gamble.

yaml

# Step 1: Add SBOM generation to the existing GitHub Actions workflow
# Append these jobs to .github/workflows/build-scan-push.yml
# after the existing push and Cosign signing steps

      # ── SBOM Generation and Attachment ─────────────────────────────────
      - name: Install Syft for SBOM generation
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        run: |
          curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh             | sh -s -- -b /usr/local/bin v1.0.1
          syft version

      - name: Generate CycloneDX SBOM
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        env:
          ECR_REGISTRY: ${{ steps.ecr_login.outputs.registry }}
        run: |
          # Generate SBOM from the pushed image (pulls from ECR for accuracy)
          syft $ECR_REGISTRY/ipl-scorecard@${{ env.IMAGE_DIGEST }}             --output cyclonedx-json=ipl-sbom.cyclonedx.json             --output spdx-json=ipl-sbom.spdx.json

          # Report component count for audit visibility
          COMPONENT_COUNT=$(python3 -c "
          import json
          with open('ipl-sbom.cyclonedx.json') as f: sbom = json.load(f)
          print(len(sbom.get('components', [])))
          ")
          echo "SBOM generated: $COMPONENT_COUNT components" >> $GITHUB_STEP_SUMMARY

      - name: Attach SBOM to image digest as OCI artifact
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        env:
          ECR_REGISTRY: ${{ steps.ecr_login.outputs.registry }}
        run: |
          # Attach CycloneDX SBOM to the image in ECR as an OCI referrer
          cosign attach sbom             --sbom ipl-sbom.cyclonedx.json             --type cyclonedx             $ECR_REGISTRY/ipl-scorecard@${{ env.IMAGE_DIGEST }}

          # Verify the SBOM referrer was recorded
          cosign tree $ECR_REGISTRY/ipl-scorecard@${{ env.IMAGE_DIGEST }}

      - name: Attach vulnerability scan as attestation
        if: github.ref == 'refs/heads/main' && github.event_name == 'push'
        env:
          ECR_REGISTRY: ${{ steps.ecr_login.outputs.registry }}
        run: |
          # Generate a vulnerability scan report in Cosign's attestation format
          trivy image             --format cosign-vuln             --output vuln-attestation.json             $ECR_REGISTRY/ipl-scorecard@${{ env.IMAGE_DIGEST }}

          # Attach as a signed attestation (not just an attached artifact)
          cosign attest --yes             --predicate vuln-attestation.json             --type vuln             $ECR_REGISTRY/ipl-scorecard@${{ env.IMAGE_DIGEST }}

Step 2 — Core Logic

Add the SLSA provenance generation step using the `slsa-github-generator` reusable workflow, and update the Kyverno ClusterPolicy from Audit to Enforce mode. The SLSA provenance generator runs as a separate GitHub Actions job that calls a reusable workflow maintained by the Sigstore project, producing a SLSA Level 3 provenance attestation that records the source repository, commit SHA, workflow definition, and builder identity. Switching Kyverno from Audit to Enforce is a one-field change that activates the enforcement gate — this step should only be done after verifying that the current workflow produces valid signatures and attestations for every commit on the main branch, which can be confirmed by checking the Kyverno Audit policy report for zero violations.

Analogy🏏Cricket

🏏 Think of it like cricket: once the practice match proves the plans work, the team steps into the official fixture — and everything must now count for real. Just as the practice-game scorecard is torn up so the official scorers start a fresh book, the staging certificate Secret is deleted so cert-manager issues a fresh production certificate rather than serving the old untrusted one. Just as the umpires formally verify the match ball and playing conditions before play begins, you verify the production certificate against the system certificate store — curl without -k must succeed. Then comes rehearsing the handover: just as a club renews a key player's contract well before it expires so there is never a day without him under contract, patching renewBefore to 89 days forces cert-manager to renew the 90-day certificate almost immediately, proving rotation happens automatically long before expiry. The payoff: trusted TLS in production plus demonstrated automatic renewal — no midnight expiry emergencies.

yaml

# Step 2: SLSA provenance + Kyverno policy update
# Full updated workflow: .github/workflows/build-scan-push.yml

# ── Add SLSA provenance generation job ─────────────────────────────────────
# This is a SEPARATE job that calls the slsa-github-generator reusable workflow
# It MUST be a separate job due to how the generator accesses the OIDC token

  # (separate from the build-scan-push job)
  slsa-provenance:
    needs: [build-scan-push]   # runs after the image is pushed
    permissions:
      id-token: write           # for OIDC token used in signing
      contents: read
      actions: read             # for accessing workflow run information
    uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v1.10.0
    with:
      image: 123456789.dkr.ecr.ap-south-1.amazonaws.com/ipl-scorecard
      digest: ${{ needs.build-scan-push.outputs.image_digest }}
      registry-username: AWS
    secrets:
      registry-password: ${{ secrets.ECR_PASSWORD }}

# ── Update the Kyverno ClusterPolicy to Enforce mode ──────────────────────
# After verifying zero Audit violations, apply the updated policy

# kyverno-ipl-scorecard-policy.yaml — production enforce mode
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-ipl-scorecard-signature
spec:
  validationFailureAction: Enforce   # CHANGED from Audit to Enforce
  background: true
  rules:
    - name: check-image-signature
      match:
        any:
          - resources: {kinds: [Pod]}
      verifyImages:
        - imageReferences:
            - "123456789.dkr.ecr.ap-south-1.amazonaws.com/ipl-scorecard:*"
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/org/ipl-scorecard/.github/workflows/build-scan-push.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"
          attestations:
            - predicateType: https://slsa.dev/provenance/v0.2
              conditions:
                - all:
                    - key: "{{ predicate.invocation.configSource.uri }}"
                      operator: Equals
                      value: "git+https://github.com/org/ipl-scorecard@refs/heads/main"
            - predicateType: https://cosign.sigstore.dev/attestation/vuln/v1
              conditions:
                - all:
                    - key: "{{ predicate.scanner.name }}"
                      operator: Equals
                      value: "Trivy"   # must have been scanned by Trivy

# Apply the Enforce policy
kubectl apply -f kyverno-ipl-scorecard-policy.yaml

# Verify: confirm current production deployment still passes
kubectl get cpol require-ipl-scorecard-signature -o yaml | grep validationFailureAction

Step 3 — Integration & Enhancement

Verify the complete supply chain from end to end: confirm all OCI referrers are present for the latest image, verify the Kyverno policy blocks an unsigned image, and use `cosign verify-attestation` to inspect the SLSA provenance and vulnerability attestation contents. This verification sequence traces the complete chain of trust from the CI signing event through the transparency log to the admission webhook, confirming that every link in the chain is intact and that the enforcement gate correctly distinguishes between signed and unsigned images.

Analogy🏏Cricket

🏏 Think of it like cricket: on the eve of a tournament the venue runs a full walk-through — a spectator's journey from the car park, through the ticket gate, to the correct stand, with stewards redirecting anyone who wanders toward the wrong entrance. Just as that walk-through exercises every link in the chain, the end-to-end check verifies DNS resolution, the TLS handshake with the production certificate, routing to the correct backend service, and the HTTPS redirect that steers plain-HTTP stragglers to the secure entrance. Just as gate staff read the stand printed on each ticket and route spectators accordingly, host-based routing reads the hostname and sends pgAdmin traffic to its own backend while API traffic goes to the API. And just as an all-venue tournament pass admits its holder to every ground without a separate ticket per stadium, the wildcard certificate covers both subdomains with a single credential. The payoff: one entry point, correctly securing and routing every kind of visitor.

bash

# End-to-end supply chain verification

ECR="123456789.dkr.ecr.ap-south-1.amazonaws.com"
IMAGE="$ECR/ipl-scorecard:latest"
DIGEST=$(docker inspect $IMAGE --format '{{index .RepoDigests 0}}' | cut -d@ -f2)

# ── 1. Inspect the full OCI referrer graph ────────────────────────────────
cosign tree $IMAGE
# Expected tree:
# └── sha256:abc123 (ipl-scorecard:latest)
#     ├── sha256:def456 (application/vnd.dev.cosign.artifact.sig.v1+json — signature)
#     ├── sha256:ghi789 (application/vnd.cyclonedx — SBOM attachment)
#     ├── sha256:jkl012 (application/vnd.dev.cosign.attestation.v1+json — SLSA provenance)
#     └── sha256:mno345 (application/vnd.dev.cosign.attestation.v1+json — vuln scan)

# ── 2. Verify Cosign signature ────────────────────────────────────────────
cosign verify   --certificate-identity-regexp "https://github.com/org/ipl-scorecard.*"   --certificate-oidc-issuer "https://token.actions.githubusercontent.com"   $ECR/ipl-scorecard@$DIGEST
echo "Signature verification: PASS ✓"

# ── 3. Inspect SLSA provenance attestation ────────────────────────────────
cosign verify-attestation   --certificate-identity-regexp "https://github.com/org/ipl-scorecard.*"   --certificate-oidc-issuer "https://token.actions.githubusercontent.com"   --type slsaprovenance   $ECR/ipl-scorecard@$DIGEST   | jq '.payload | @base64d | fromjson | {
      builder: .predicate.builder.id,
      source: .predicate.invocation.configSource.uri,
      commit: .predicate.invocation.configSource.digest.sha1,
      buildTime: .predicate.metadata.buildFinishedOn
    }'
# Expected: source and commit link back to the correct repo and commit

# ── 4. Test Kyverno enforcement: unsigned image must be blocked ───────────
# Push an unsigned test image (skip the signing step)
docker tag python:3.11-slim $ECR/ipl-scorecard:unsigned-test
docker push $ECR/ipl-scorecard:unsigned-test

# Attempt to create a Pod using the unsigned image — must be blocked
kubectl run unsigned-pod   --image=$ECR/ipl-scorecard:unsigned-test   --dry-run=server  # server-side dry run runs admission webhooks
# Expected: Error from server (Forbidden): pods "unsigned-pod" is forbidden:
#   policy require-ipl-scorecard-signature/check-image-signature:
#   image verification failed for .../ipl-scorecard:unsigned-test
echo "Unsigned image blocked by Kyverno: PASS ✓"

# ── 5. Confirm signed image deploys successfully ──────────────────────────
kubectl run signed-pod   --image=$ECR/ipl-scorecard:latest   --dry-run=server
# Expected: pod/signed-pod created (dry run) — signature verified
echo "Signed image admitted by Kyverno: PASS ✓"

# ── 6. Review Kyverno policy report for any unexpected violations ─────────
kubectl get policyreport -A -o json   | jq '.items[].results[] | select(.result != "pass") | {policy, resource, message}'
# Expected: empty output — no violations in the cluster

Step 4 — Testing & Verification

Complete the lab verification checklist, which confirms every supply chain property independently. The checklist is designed so that each item produces an unambiguous pass or fail: the Cosign verify command exits 0 or non-zero, the Kyverno dry-run admission either succeeds or returns a Forbidden error, the SBOM component count is a non-zero integer. A lab that passes all checklist items has demonstrated the complete supply chain integrity chain from source commit to admission enforcement, ready for the M5 module's deeper treatment of Kubernetes RBAC and Pod Security Standards.

Analogy🏏Cricket

🏏 Think of it like cricket: at the midpoint of a season, a good head coach doesn't just check the points table — he traces how every department produced it: the academy that developed the players, the selectors who picked the squad, and the matchday operations that got the team onto the field. Just as that review makes the whole club visible as one system rather than isolated departments, the M1–M4 architecture summary traces each component's role — the container image layer where the application is built and packaged, the Kubernetes workload layer where Deployments, StatefulSets, and autoscaling run it, and the external access layer where Ingress and TLS expose it to the world. Just as ticking the final checklist confirms match-readiness, completing the lab checklist confirms every layer actually works together. The payoff: you can explain the entire platform end to end — the mark of real understanding, and the foundation M5's security work builds on.

bash

# Final verification checklist

echo "=== M2 Lab — Supply Chain Security Verification ==="

# 1. SBOM attached as OCI referrer
echo -n "1. SBOM attached to image:              "
cosign tree "$ECR/ipl-scorecard:latest" 2>/dev/null | grep -q "cyclonedx"   && echo "PASS ✓" || echo "FAIL ✗"

# 2. Cosign signature present and verifiable
echo -n "2. Cosign signature valid:              "
cosign verify   --certificate-identity-regexp "https://github.com/org/ipl-scorecard.*"   --certificate-oidc-issuer "https://token.actions.githubusercontent.com"   "$ECR/ipl-scorecard:latest" > /dev/null 2>&1   && echo "PASS ✓" || echo "FAIL ✗"

# 3. SLSA provenance attestation present and points to correct repo
echo -n "3. SLSA provenance attestation valid:   "
cosign verify-attestation   --certificate-identity-regexp "https://github.com/org/ipl-scorecard.*"   --certificate-oidc-issuer "https://token.actions.githubusercontent.com"   --type slsaprovenance   "$ECR/ipl-scorecard:latest" 2>/dev/null   | jq -e '.payload | @base64d | fromjson | .predicate.invocation.configSource.uri'   | grep -q "github.com/org/ipl-scorecard"   && echo "PASS ✓" || echo "FAIL ✗"

# 4. Vulnerability scan attestation present
echo -n "4. Vuln scan attestation attached:      "
cosign tree "$ECR/ipl-scorecard:latest" 2>/dev/null | grep -q "vuln"   && echo "PASS ✓" || echo "FAIL ✗"

# 5. Kyverno blocks unsigned image
echo -n "5. Kyverno blocks unsigned image:       "
kubectl run test-unsigned   --image="$ECR/ipl-scorecard:unsigned-test"   --dry-run=server 2>&1 | grep -q "forbidden"   && echo "PASS ✓" || echo "FAIL ✗"

# 6. Kyverno admits signed image
echo -n "6. Kyverno admits signed image:         "
kubectl run test-signed   --image="$ECR/ipl-scorecard:latest"   --dry-run=server > /dev/null 2>&1   && echo "PASS ✓" || echo "FAIL ✗"

# 7. Zero Kyverno policy violations in cluster
echo -n "7. No unexpected policy violations:     "
VIOLATIONS=$(kubectl get policyreport -A -o json 2>/dev/null   | python3 -c "
import sys, json
data = json.load(sys.stdin)
n = sum(1 for item in data.get('items',[]) for r in item.get('results',[]) if r.get('result') != 'pass')
print(n)
")
[ "$VIOLATIONS" -eq "0" ] && echo "PASS ✓ (0 violations)" || echo "FAIL ✗ ($VIOLATIONS violations)"

echo "=== Lab complete — supply chain integrity chain verified ==="

Warning: The `slsa-github-generator` reusable workflow must be called from a separate GitHub Actions job, not from a step within the existing build job. The generator requires isolated access to the GitHub Actions OIDC token to produce a Level 3 provenance attestation; running it from a shared job context would downgrade the attestation to Level 2 because the provenance could theoretically be influenced by other steps in the same job. The `needs:` key that makes the provenance job depend on the build job is not optional — the build job must complete successfully and output the image digest before the provenance generator can reference it. Attempting to inline the slsa-github-generator as a step will silently produce an invalid attestation that fails SLSA Level 3 verification.

Extension Challenge: Implement a Kubernetes `NetworkPolicy` that restricts the `ipl-scorecard` pods to only accept inbound traffic from Nginx pods (using pod label selectors) and only allow outbound traffic to the `ipl_postgres` and `ipl_redis` pods. This adds the Kubernetes-native equivalent of the Compose multi-network micro-segmentation from the M2 Exercise — the same security boundary, implemented at the CNI layer rather than the Docker bridge layer. Compare the Compose network declaration to the Kubernetes NetworkPolicy specification line by line as a preview of the M3 module's coverage of Kubernetes networking primitives.

Implement supply chain controls incrementally — generate SBOM, add signing, add provenance, then enforce — verifying each layer independently before adding the next to avoid compound failures that obscure root causes.
SBOM attachment as an OCI referrer co-locates the software inventory with the image in the registry, making it queryable without external infrastructure and ensuring it travels with the image through registry mirrors.
The SLSA provenance generator must run in a separate GitHub Actions job with isolated OIDC token access to produce Level 3 provenance — inlining it as a step in the build job silently downgrades the attestation.
Always deploy Kyverno policies in Audit mode first and monitor the policy report for violations before switching to Enforce — switching directly to Enforce on an unsigned deployment will immediately block all pods.
The full supply chain trust chain — source commit → OIDC-bound signing → Rekor log entry → Kyverno admission verification — is end-to-end verifiable offline using Rekor's root of trust without contacting any external service.
Use `cosign tree <image>` to inspect the complete OCI referrer graph for an image, confirming that signatures, SBOM attachments, and attestations are all present and correctly associated with the image digest.