Capstone — Production Container Platform

This capstone is the culmination of the entire course: you will design and build a complete production container platform for a cricket scoring application, integrating every major theme — image building and optimisation, multi-container orchestration, networking and isolation, persistent storage, security hardening, observability, resource management, a CI/CD pipeline, and registry strategy. The platform runs a microservices-style scoring system behind a reverse proxy, with isolated data tiers, secrets, health-gated startup, full monitoring, and an automated pipeline that builds, tests, scans, signs and promotes images by digest. This is the kind of end-to-end system that demonstrates genuine production competence and makes a standout portfolio piece, because it shows you can not only run a container but operate a secure, observable, automatically-delivered platform. By the end you will have assembled, from first principles, the platform that everything in this course was building toward.

Learning Objectives

• Integrate image optimisation, multi-container orchestration, networking, storage, security, observability and CI/CD into one coherent platform.
• Design a layered, isolated architecture where only an edge proxy is publicly reachable and data tiers are fully internal.
• Harden every service with non-root users, dropped capabilities, read-only filesystems, secrets and resource limits.
• Build a complete CI/CD pipeline that builds once, tests, scans, signs, and promotes images by digest across environments.
• Instrument the platform end to end with health checks, structured logs, metrics and dashboards for full observability.
• Demonstrate self-healing, graceful shutdown and reproducible deployment so the platform is operable, not just runnable.

Technical Requirements

• A scorecard API and a separate accounts service, each owning its own PostgreSQL database, collaborating only over the network.
• A Redis cache for live run-rate data and a reverse proxy as the single public entry point on one published port.
• Frontend and backend networks segmenting public traffic from the data tier, with databases and cache having no host-published ports.
• Database credentials delivered as mounted secrets, with all non-secret configuration injected via env files at deploy time.
• Health checks on every long-running service and conditional dependencies enforcing correct, race-free startup order.
• Per-service resource requests and limits, non-root users, dropped capabilities, read-only root filesystems and no-new-privileges.
• A Prometheus and Grafana observability stack scraping metrics from the services, plus structured logging to stdout.
• A CI/CD pipeline that builds once with caching, runs tests, scans for CVEs, signs the image, and promotes by digest to a production project.

Architecture & Design

The platform is a layered, defence-in-depth system. At the edge, a single Nginx reverse proxy publishes one port to the world and routes requests inward; nothing else is publicly reachable. The proxy and the two application services — scorecard-api and accounts-api — share a frontend network, while those services, their two databases, the cache and a migration job share a backend network, so the databases and cache have no exposure outside the stack. Each application service owns its own database and never touches the other's, collaborating only over the network, which keeps the services independently deployable. Startup is ordered by readiness: each database exposes a health check, migration jobs wait for database health and complete before the APIs start, and the proxy waits for API health. Configuration flows in through env files and a mounted secret for credentials. Every service is hardened — non-root, capability-dropped, read-only with tmpfs scratch, resource-limited — and instrumented with a metrics endpoint and structured logs. A Prometheus and Grafana pair on the backend network scrapes and visualises the metrics. Above the running system sits the delivery layer: a CI/CD pipeline builds images once, tests and scans them, signs the digest, and promotes that exact digest from a quarantine project to production. This architecture composes every course concept into one secure, observable, automatically-delivered platform.

# Platform structure — services, delivery, and observability
cricket-platform/
├── .github/workflows/cicd.yml      # build-once -> test -> scan -> sign -> promote
├── compose.yaml                    # the full runtime platform
├── .env / .env.example             # non-secret config (real one git-ignored)
├── secrets/
│   ├── scorecard_db_password.txt
│   └── accounts_db_password.txt
├── scorecard/  { Dockerfile, app.py, migrate.py, requirements.txt }
├── accounts/   { Dockerfile, app.py, migrate.py, requirements.txt }
├── proxy/      { nginx.conf }      # routes /api/scores and /api/users
└── observability/ { prometheus.yml, grafana/ }

# Network + exposure model (only 'edge' is public):
#   internet -> edge(:8080) -> [frontend] -> scorecard-api, accounts-api
#                                  |              |            |
#                              [backend] -> scorecard-db, accounts-db, score-cache,
#                                            prometheus, grafana, migrations

Phase 1 — Core Implementation

Phase 1 builds the runtime platform: the two hardened, instrumented application services, their databases, the cache, the migration jobs, and the reverse proxy, wired together in Compose with network segmentation, secrets, health checks and conditional dependencies. This is the heart of the platform — a secure, isolated, health-gated multi-service system. Each service reads its database password from a mounted secret, exposes /healthz and /metrics, runs non-root with dropped capabilities and a read-only filesystem, and carries resource limits. Getting this phase right means the platform runs and defends itself.

# compose.yaml (core excerpt) — segmented, hardened, health-gated platform
services:
  edge:                                  # only public entry point
    image: nginx:1.27
    ports: ["${EDGE_PORT:-8080}:80"]
    volumes: ["./proxy/nginx.conf:/etc/nginx/nginx.conf:ro"]
    networks: [frontend]
    read_only: true
    tmpfs: ["/var/cache/nginx", "/var/run"]
    cap_drop: ["ALL"]; cap_add: ["NET_BIND_SERVICE"]
    security_opt: ["no-new-privileges:true"]
    depends_on:
      scorecard-api: { condition: service_healthy }
      accounts-api:  { condition: service_healthy }

  scorecard-api:                         # owns scorecard-db
    image: ghcr.io/cricket/scorecard@${SCORECARD_DIGEST}   # pinned by digest
    networks: [frontend, backend]
    env_file: [./.env]
    environment: { DB_PASSWORD_FILE: /run/secrets/scorecard_db_password, DB_HOST: scorecard-db, CACHE_HOST: score-cache }
    secrets: [scorecard_db_password]
    read_only: true; tmpfs: ["/tmp"]
    cap_drop: ["ALL"]; security_opt: ["no-new-privileges:true"]
    deploy: { resources: { limits: { cpus: '0.5', memory: 256M }, reservations: { cpus: '0.25', memory: 128M } } }
    healthcheck: { test: ["CMD","curl","-fsS","http://localhost:5000/healthz"], interval: 10s, retries: 3, start_period: 15s }
    depends_on:
      scorecard-db:      { condition: service_healthy }
      scorecard-migrate: { condition: service_completed_successfully }

  scorecard-db:                          # internal only, durable volume
    image: postgres:16
    networks: [backend]
    environment: { POSTGRES_USER: admin, POSTGRES_DB: scores, POSTGRES_PASSWORD_FILE: /run/secrets/scorecard_db_password }
    secrets: [scorecard_db_password]
    volumes: ["scorecard-data:/var/lib/postgresql/data"]
    healthcheck: { test: ["CMD-SHELL","pg_isready -U admin -d scores"], interval: 5s, retries: 5, start_period: 10s }

  # accounts-api + accounts-db + accounts-migrate mirror the scorecard trio,
  # each owning its OWN database (no shared database).
  score-cache: { image: redis:7, networks: [backend], security_opt: ["no-new-privileges:true"] }

networks: { frontend: {}, backend: {} }
volumes: { scorecard-data: {}, accounts-data: {} }
secrets:
  scorecard_db_password: { file: ./secrets/scorecard_db_password.txt }
  accounts_db_password:  { file: ./secrets/accounts_db_password.txt }

Phase 2 — Feature Completion

Phase 2 adds the observability stack and the CI/CD pipeline, completing the platform's operational and delivery layers. Prometheus and Grafana join the backend network, with Prometheus scraping each service's /metrics endpoint and Grafana visualising request rates, run rate and resource usage. The pipeline builds each service image once with caching, runs unit and integration tests, scans for vulnerabilities, signs the digest, and promotes that exact digest from a quarantine project to production. Now the platform is not only running and secure but observable and automatically, verifiably delivered.

# .github/workflows/cicd.yml — build once, test, scan, sign, promote by digest
name: cricket-platform-cicd
on: { push: { branches: [main] }, pull_request: {} }
jobs:
  deliver:
    runs-on: ubuntu-latest
    permissions: { contents: read, packages: write, id-token: write }
    strategy: { matrix: { svc: [scorecard, accounts] } }   # both services
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - name: Build ONCE into quarantine (with cache)
        id: build
        uses: docker/build-push-action@v6
        with:
          context: ./${{ matrix.svc }}
          push: true
          tags: ghcr.io/cricket/quarantine/${{ matrix.svc }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
          sbom: true
          provenance: true
      - name: Test inside the built image
        run: docker run --rm ghcr.io/cricket/quarantine/${{ matrix.svc }}:${{ github.sha }} python -m pytest -q
      - name: Scan (fail on fixable HIGH/CRITICAL)
        run: docker run --rm aquasec/trivy image --exit-code 1 --ignore-unfixed --severity HIGH,CRITICAL ghcr.io/cricket/quarantine/${{ matrix.svc }}:${{ github.sha }}
      - name: Sign the digest (keyless via OIDC)
        if: github.ref == 'refs/heads/main'
        run: cosign sign --yes ghcr.io/cricket/quarantine/${{ matrix.svc }}@${{ steps.build.outputs.digest }}
      - name: Promote the SAME digest to production (main only)
        if: github.ref == 'refs/heads/main'
        run: |
          docker buildx imagetools create \
            --tag ghcr.io/cricket/production/${{ matrix.svc }}:${{ github.sha }} \
            ghcr.io/cricket/quarantine/${{ matrix.svc }}@${{ steps.build.outputs.digest }}

Phase 3 — Polish & Production Readiness

Phase 3 proves the platform is operable, not merely assembled. Add restart policies and graceful SIGTERM handling so services self-heal and drain cleanly on deploy, provision a Grafana dashboard from a file so monitoring is reproducible, and require signature verification before any production image runs. Then write an end-to-end verification script that brings the platform up, exercises both services through the proxy, confirms the databases are isolated, checks metrics are flowing, kills a service to demonstrate self-healing, and verifies an image signature. Passing this script is concrete evidence that every layer works together under realistic conditions.

# verify_platform.sh — end-to-end proof the whole platform works
#!/usr/bin/env bash
set -euo pipefail
docker compose up -d

# 1) Both services reachable ONLY through the public proxy:
until curl -fsS localhost:8080/healthz >/dev/null; do sleep 2; done
curl -fsS -X POST localhost:8080/api/scores \
  -H 'Content-Type: application/json' \
  -d '{"batsman":"Rohit Sharma","runs":91,"overs":11}'
curl -fsS localhost:8080/api/users/health

# 2) Data tiers are isolated (no host-published db/cache ports):
! nc -z -w2 localhost 5432 2>/dev/null && echo 'scorecard-db isolated'
! nc -z -w2 localhost 6379 2>/dev/null && echo 'cache isolated'

# 3) Observability: Prometheus targets up, metrics flowing:
curl -s localhost:9090/api/v1/targets | grep -o '"health":"up"' | wc -l

# 4) Self-healing: kill the scorecard API, confirm it returns healthy:
docker kill "$(docker compose ps -q scorecard-api)"
sleep 8; curl -fsS localhost:8080/healthz && echo 'self-healed'

# 5) Supply chain: verify the production image signature:
cosign verify ghcr.io/cricket/production/scorecard:"$GIT_SHA" \
  --certificate-identity-regexp '.*' --certificate-oidc-issuer-regexp '.*' \
  && echo 'signature verified'

docker compose down -v

Evaluation Rubric

• Architecture: layered and isolated — only the edge proxy is publicly reachable, and databases and cache have no host-published ports.
• Service independence: scorecard and accounts each own a separate database and collaborate only over the network, with no shared database.
• Security: every service runs non-root with cap_drop ALL, no-new-privileges, read-only root filesystem and tmpfs, with credentials as mounted secrets.
• Reliability: health checks plus conditional dependencies give race-free startup, restart policies self-heal, and SIGTERM is handled gracefully.
• Observability: Prometheus scrapes each service's metrics, Grafana visualises them from a provisioned dashboard, and services log structured JSON to stdout.
• Delivery: the pipeline builds once with caching, tests, scans, signs the digest and promotes that exact digest to production, with signatures verified before run.
• Verification: an end-to-end script demonstrates traffic, isolation, metrics, self-healing and signature verification on a fresh bring-up.