Practice — full DevSecOps pipeline: SAST, SCA, sign and Kyverno — CI/CD, GitOps, DevSecOps & Observability

What You'll Build

In this exercise you will extend the india-squad FastAPI application's CI pipeline with a complete DevSecOps security layer. The finished pipeline has four sequential security gates: Semgrep SAST scanning with a custom rule for hardcoded credentials, Snyk SCA scanning that blocks on upgradeable HIGH CVEs, Trivy container scanning for image vulnerabilities, and Cosign keyless image signing. The build and sign job only runs after both the SAST and SCA jobs pass.

You will intentionally introduce a hardcoded credential and a vulnerable dependency, run the pipeline to see the failures, fix both issues, and confirm the pipeline turns green. You will then intentionally re-introduce the SAST failure to verify the Build-Scan-Sign job is correctly blocked by the SAST gate.

Analogy🏏Cricket

Think of it like cricket: Picture setting up a remote ground management system for a cricket ground in a new city. First, the infrastructure must be installed: the pitch sensors, the scoreboard network, and the broadcast uplink—equivalent to installing ArgoCD on the EKS cluster. Then, the venue configuration must be committed to the central venue management database: the pitch dimensions, the lighting schedule, the boundary positions—equivalent to committing the Kubernetes manifests to the GitOps repository. Then, the venue must be registered with the central management system, which then automatically enforces the declared configuration at the ground—equivalent to creating the ArgoCD Application that connects the repository to the cluster. When a ground manager moves a boundary rope by hand, the sensors detect the drift and alert the management system to restore the declared position—equivalent to ArgoCD detecting and reverting the manual replica scale. This reveals why the lab sequence matters: you cannot verify GitOps until all three components—operator, repository, and Application—are connected and working together.

Prerequisites

The india-squad-helm project from Module 2 Exercise 13, with a working GitHub repository and GitHub Actions enabled.
A Semgrep account (free tier) or the SEMGREP_APP_TOKEN secret set in the repository—without it Semgrep runs in open-source mode against the OWASP ruleset only.
A Snyk account (free tier) with the SNYK_TOKEN secret set in the repository settings under Settings → Secrets → Actions.
The ECR_ROLE_ARN secret configured from Module 1 Lab 7, since the build-scan-sign job pushes the image to ECR using OIDC federation.
Cosign CLI installed locally for the verification step: brew install cosign on macOS or the Linux binary from github.com/sigstore/cosign/releases.

Setup & Introduce Intentional Vulnerabilities

Start from the existing application, add a Semgrep custom rule, intentionally introduce a hardcoded credential into the Python code, and add a vulnerable dependency to requirements.txt. These deliberate vulnerabilities will trigger the SAST and SCA gates in Step 1 and Step 2, demonstrating that the pipeline correctly detects and blocks them before the image is built.

Analogy🏏Cricket

🏏 Think of it like cricket: Before you trust a new stadium screening, you deliberately send a decoy through carrying a banned item to confirm the scanners actually catch it — exactly as this exercise plants a hardcoded credential in the Python code, adds a known-vulnerable dependency to requirements.txt, and writes a custom Semgrep rule so the SAST and SCA gates have something real to catch. Just as a screening you never test against a real threat gives false confidence, a pipeline you never feed a real vulnerability might be silently passing everything. Just as the decoy proves the metal detector triggers and the bag scanner flags the item, the planted secret proves Semgrep fires and the bad package proves the dependency scanner fires. The payoff: intentionally introducing the flaws is how you prove the gates genuinely block bad code before the image is ever built and pushed.

bash

# Starting point: the india-squad FastAPI app from Module 2's exercise.
# We extend the existing CI pipeline to add SAST, SCA, image signing.

cd india-squad-helm   # from the Helm chart exercise

# Ensure the workflow directory exists
mkdir -p .github/workflows .semgrep

# Create a Semgrep custom rule for the project
cat > .semgrep/india-squad.yaml << 'EOF'
rules:
  - id: hardcoded-squad-secret
    message: 'Hardcoded credential detected. Use environment variables.'
    severity: ERROR
    languages: [python]
    patterns:
      - pattern: $VAR = '...'
      - metavariable-regex:
          metavariable: $VAR
          regex: '(?i)(key|secret|password|token)'
EOF

# Add a vulnerability for Semgrep to find (we will fix it in Step 2)
cat >> app/main.py << 'APPEOF'

# TODO: remove before production
ROHIT_API_KEY = 'AKIAIOSFODNN7EXAMPLE'
APPEOF

# Add a vulnerable dependency (we will fix it in Step 3)
echo 'cryptography==38.0.0' >> requirements.txt
# cryptography 38.0.0 has known HIGH CVEs

git add . && git commit -m 'chore: setup devsecops exercise base'

Step 1 — SAST Gate with Semgrep

Add the Semgrep job to the workflow. The pipeline will fail on the first push because the hardcoded `ROHIT_API_KEY` triggers the custom rule. Fix the issue by removing the hardcoded credential, commit the fix, and push again. The Semgrep step should now pass, demonstrating that the SAST gate correctly catches and blocks the vulnerability.

Analogy🏏Cricket

Think of it like cricket: The Semgrep step is the fitness assessment. The candidate—the code commit—must pass the fitness test before advancing. When the hardcoded credential is detected, the candidate fails the fitness test immediately, and no further assessment is performed. Fixing the credential is the equivalent of the candidate completing the required training and retesting. The SARIF upload with `if: always()` is the fitness assessment record: it is filed regardless of pass or fail, providing an auditable record of every assessment outcome.

bash

# Step 1: Add SAST (Semgrep) as a blocking CI gate.

cat > .github/workflows/devsecops.yml << 'WFEOF'
name: india-squad-devsecops

on:
  push:    { branches: [main] }
  pull_request: { branches: [main] }

permissions:
  contents:       read
  security-events: write

env:
  REGISTRY: 123456789.dkr.ecr.ap-south-1.amazonaws.com
  IMAGE:    india-squad-api

jobs:
  sast:
    name: SAST — Semgrep
    runs-on: ubuntu-latest
    container: { image: semgrep/semgrep }
    steps:
      - uses: actions/checkout@v4
      - name: Run Semgrep
        run: |
          semgrep ci \
            --config p/python \
            --config .semgrep/ \
            --sarif --output semgrep.sarif \
            --error
      - name: Upload SARIF
        if: always()
        uses: github/codeql-action/upload-sarif@v3
        with: { sarif_file: semgrep.sarif, category: semgrep }
WFEOF

git add .github/workflows/devsecops.yml
git commit -m 'ci: add Semgrep SAST gate'
git push

# The pipeline will FAIL because of the hardcoded API key we added.
# Fix it by removing the hardcoded credential:
sed -i '/ROHIT_API_KEY/d' app/main.py
git add app/main.py
git commit -m 'fix(security): remove hardcoded API key'
git push
# Pipeline should now pass the Semgrep step.

Step 2 — SCA Gate with Snyk

Add the Snyk job to the workflow. The pipeline will fail because `cryptography==38.0.0` has HIGH CVEs with available fixes, triggering the `--fail-on=upgradeable` flag. Upgrade cryptography to the patched version, commit, and push. Both SAST and SCA should now pass in parallel—the two jobs run concurrently, and the Build-Scan-Sign job waits for both.

Analogy🏏Cricket

Think of it like cricket: Adding the Snyk job is like adding the equipment standards check to the selection process. Even if a candidate has perfect technique—SAST passes—they cannot play with recalled equipment—a vulnerable dependency. The `--fail-on=upgradeable` flag is like the recall policy: the equipment is only rejected if a replacement is available. A recalled item with no available replacement is flagged but not grounds for disqualification yet—the team manages the risk while waiting for the replacement batch. The fix—upgrading cryptography—is the equivalent of replacing the recalled kit with the approved replacement batch.

bash

# Step 2: Add SCA (Snyk) gate — fix the vulnerable cryptography dependency.

# Add Snyk job to the workflow (append to the existing jobs section):
cat >> .github/workflows/devsecops.yml << 'WFEOF'

  sca:
    name: SCA — Snyk
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - name: Snyk SCA
        uses: snyk/actions/python@master
        env:
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
        with:
          args: --severity-threshold=high --fail-on=upgradeable --sarif-file-output=snyk.sarif
      - name: Upload Snyk SARIF
        if: always()
        uses: github/codeql-action/upload-sarif@v3
        with: { sarif_file: snyk.sarif, category: snyk-sca }
WFEOF

git add .github/workflows/devsecops.yml
git commit -m 'ci: add Snyk SCA gate'
git push

# The SCA step will FAIL because cryptography==38.0.0 has HIGH CVEs.
# Fix it by upgrading to the patched version:
sed -i 's/cryptography==38.0.0/cryptography>=41.0.3/' requirements.txt
git add requirements.txt
git commit -m 'fix(deps): upgrade cryptography to patch HIGH CVEs'
git push
# Both SAST and SCA should now pass.

Step 3 — Build, Trivy Scan, and Cosign Sign

Add the build-scan-sign job that runs only after SAST and SCA pass. The job builds the Docker image locally, runs Trivy to scan for container-level vulnerabilities, pushes the clean image to ECR, and signs it with Cosign using OIDC keyless signing. The `needs: [sast, sca]` dependency ensures the container is never built from code that failed either security gate.

Analogy🏏Cricket

Think of it like cricket: The build-scan-sign job is the field trial and credential issuance: only candidates who passed fitness and equipment checks participate in it. The Trivy scan checks the actual match ball—the container image—for manufacturing defects that the pre-selection checks could not catch. The Cosign signing is the official credential issuance: a cryptographic attestation attached to the specific player—the specific image digest—that certifies they completed the full selection process. Just as the credential is tied to the specific player and cannot be transferred to a different player, the Cosign signature is tied to the specific image digest and cannot be transferred to a different image.

bash

# Step 3: Add Trivy container scan + Cosign signing.

# Append build, scan, and sign jobs to the workflow:
cat >> .github/workflows/devsecops.yml << 'WFEOF'

  build-scan-sign:
    name: Build, Scan & Sign
    needs: [sast, sca]          # only run after both security gates pass
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write           # for OIDC to AWS and Sigstore Fulcio
      security-events: write
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS OIDC
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.ECR_ROLE_ARN }}
          aws-region: ap-south-1

      - id: ecr-login
        uses: aws-actions/amazon-ecr-login@v2

      - uses: docker/setup-buildx-action@v3

      - name: Build image (no push yet)
        uses: docker/build-push-action@v5
        with:
          push: false
          load: true
          tags: ${{ env.REGISTRY }}/${{ env.IMAGE }}:${{ github.sha }}
          cache-from: type=gha
          cache-to:   type=gha,mode=max

      - name: Trivy scan
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ env.REGISTRY }}/${{ env.IMAGE }}:${{ github.sha }}
          severity: HIGH,CRITICAL
          exit-code: '1'
          ignore-unfixed: true
          format: sarif
          output: trivy.sarif

      - name: Upload Trivy SARIF
        if: always()
        uses: github/codeql-action/upload-sarif@v3
        with: { sarif_file: trivy.sarif, category: trivy }

      - name: Push image
        id: push
        uses: docker/build-push-action@v5
        with:
          push: true
          tags: ${{ env.REGISTRY }}/${{ env.IMAGE }}:${{ github.sha }}
          cache-from: type=gha

      - name: Install Cosign
        uses: sigstore/cosign-installer@v3

      - name: Sign image
        run: |
          cosign sign --yes \
            ${{ env.REGISTRY }}/${{ env.IMAGE }}@${{ steps.push.outputs.digest }}
WFEOF

git add .github/workflows/devsecops.yml
git commit -m 'ci: add Trivy scan and Cosign signing'
git push

Step 4 — Verify End-to-End

Verify the complete pipeline by confirming all jobs passed, checking the GitHub Security tab for the SARIF findings from all three tools, verifying the image signature with the cosign CLI, and then deliberately re-introducing the SAST failure to confirm the build-scan-sign job is skipped when a gate fails. The revert test is the most important: it confirms that the pipeline correctly blocks the entire chain on a security failure, not just the security job itself.

Analogy🏏Cricket

🏏 Think of it like cricket: You certify the full match-day security chain by walking it end to end — every checkpoint reports clear, the incident log shows each scanner's findings, the sealed equipment verifies as genuine — and then you deliberately smuggle a banned item back in to confirm the gate actually shuts the whole operation down. That is exactly the pipeline verification: confirm all jobs passed, read the SARIF findings from all three tools in the Security tab, verify the image signature with the cosign CLI, then re-introduce the SAST failure to confirm the build-scan-sign job is skipped. Just as re-testing the failing checkpoint is the most important proof of all, the revert test that a failed gate blocks the build is what really matters. The payoff: proving the pipeline halts on a real failure is what makes its green runs trustworthy rather than merely reassuring.

bash

# Step 4: Verify the complete DevSecOps pipeline end-to-end.

# After the pipeline completes successfully, verify:

# 1. All three security jobs passed
# Open: https://github.com/<org>/india-squad-helm/actions
# Expected: SAST, SCA, and Build-Scan-Sign all green.

# 2. GitHub Security tab shows the SARIF findings
# Open: https://github.com/<org>/india-squad-helm/security/code-scanning
# Expected: findings from Semgrep, Snyk, and Trivy visible.

# 3. Image is signed and verifiable
IMAGE='123456789.dkr.ecr.ap-south-1.amazonaws.com/india-squad-api:<sha>'
cosign verify \
  --certificate-identity-regexp='https://github.com/<org>/india-squad-helm' \
  --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
  $IMAGE
# Expected: Verification for ...: true

# 4. Test that a deliberate vulnerability blocks the pipeline
echo "VIRAT_SECRET = 'hardcoded_token_12345'" >> app/main.py
git add app/main.py && git commit -m 'test: introduce deliberate SAST failure'
git push
# Expected: SAST job fails; Build-Scan-Sign job is skipped (needs: [sast, sca])

# Revert the test commit:
git revert HEAD --no-edit && git push

Warning: The `needs: [sast, sca]` dependency in the build-scan-sign job prevents the job from running when SAST or SCA fails. However, GitHub Actions also skips the job—without failing it—when an upstream job is skipped (for example, if the `sast` job is conditionally skipped by an `if:` expression). If you add conditional logic to the SAST or SCA jobs in the future, verify that the build-scan-sign job's `needs` dependency still triggers a failure rather than a skip when the upstream jobs do not run. Use `if: needs.sast.result == 'success' && needs.sca.result == 'success'` on the build-scan-sign job to explicitly require success rather than relying on the default skip-on-skip behaviour.

Extension Challenge: Add a Kyverno policy in the GitOps repository that requires the Cosign signature from this pipeline before any image can be deployed to squad-prod. Deploy the Kyverno policy via ArgoCD and verify that a manually crafted pod referencing an unsigned image is rejected by the admission webhook. Then deploy the signed image from this pipeline and confirm it is admitted. This connects the CI signing step to the runtime enforcement gate, completing the full supply chain security chain.