Advanced Practice — Secure Multi-service Stack

What You'll Build

In this exercise you will harden and instrument the cricket scorecard stack into a secure, observable multi-service deployment that brings together everything from Module 4. You will start from a working three-service stack — a scorecard API, a PostgreSQL database and a Redis cache behind an Nginx proxy — and systematically apply the advanced techniques: network segmentation that isolates the data tier, container hardening with non-root users, dropped capabilities and read-only filesystems, image scanning to catch vulnerabilities, and a Prometheus-plus-Grafana observability layer that scrapes metrics from the API. By the end you will have a stack that not only runs but defends itself against common attacks and exposes enough signals to debug and operate confidently. This is the kind of production-grade configuration that demonstrates real operational maturity rather than a toy demo.

Analogy🏏Cricket

🏏 Think of it like cricket: Just as a player's first practice session puts the fundamental skills together in sequence — take guard, play some shots, run between wickets, review — rather than in isolation, this exercise puts the Docker fundamentals together in sequence: pull, run, manage, inspect, clean up. The insight is that fluency comes from rehearsing the basics as a connected flow: the practice session links the skills into real play, exactly as this exercise links the Docker commands into a real workflow.

Prerequisites

Docker Engine and the Compose plugin installed, with the ability to run docker compose commands locally.
Completion of Module 3 (Compose multi-container apps, networks, health checks, secrets) and Module 4 lessons on security and observability.
Basic comfort editing YAML and a Dockerfile, and reading container logs with docker compose logs.
A vulnerability scanner available — docker scout (bundled) or Trivy installed — to scan the API image.
Roughly 1 GB of free memory to run the API, database, cache, proxy, Prometheus and Grafana together.

Setup & Project Structure

You will work in a single project directory containing the Compose file, the API source and Dockerfile, the Nginx and Prometheus configuration, and a secret file for the database password. Keeping configuration in version-controlled files and the real secret values git-ignored mirrors how production projects are organised. The structure separates the application code from the operational configuration so each can be reviewed independently, and it gives Prometheus a place to define what to scrape. Create the layout below before starting.

Analogy🏏Cricket

🏏 Think of it like cricket: starting this practice with official nginx and alpine images is like turning up to nets with the ground's standard-issue kit already laid out — you need install nothing of your own. Just as a coach first confirms the nets are booked and the bowling machine is switched on before a session begins, you verify Docker is working before anything else. Just as a player draws the standard bat and pads from the club store rather than crafting gear from scratch, you pull ready-made images from Docker Hub — nginx as your web-server 'all-rounder', alpine as a tiny, nimble twelfth man. Then, just as a session moves methodically from knocking-in to full-pace deliveries to fitness cool-down, you will pull images, run containers, and manage them through their full lifecycle. The payoff: a friction-free, command-line-only foundation session where every fundamental container move gets rehearsed cleanly.

bash

secure-scorecard/
├── compose.yaml                 # the full hardened, observable stack
├── .gitignore                   # ignores secrets/ and .env
├── secrets/
│   └── db_password.txt          # the DB password (git-ignored)
├── api/
│   ├── Dockerfile               # hardened, non-root image
│   ├── requirements.txt         # flask, psycopg2-binary, redis, prometheus-client
│   └── app.py                   # scorecard API + /healthz + /metrics
├── proxy/
│   └── nginx.conf               # routes / and /api to the API
└── observability/
    └── prometheus.yml           # scrape config targeting scorecard-api:5000

# Create the skeleton and the secret:
mkdir -p secure-scorecard/{api,proxy,observability,secrets}
cd secure-scorecard
printf 'superSecretMatchPassword' > secrets/db_password.txt
printf 'secrets/\n.env\n' > .gitignore

Step 1 — Foundation

Step 1 builds the hardened API image and exposes a metrics endpoint, establishing the foundation the rest of the stack secures and observes. The Dockerfile creates a dedicated non-root user so the process never runs as root, and the application reads its database password from the mounted secret file rather than an environment variable. Crucially, the API uses the prometheus client to publish a /metrics endpoint, so the observability layer added later has something real to scrape. This step puts least-privilege and instrumentation in place at the source.

Analogy🏏Cricket

🏏 Think of it like cricket: before opening a stadium you first vet the players themselves — confirm their credentials, assign each a limited-access pass, and fit them with the tracking sensors that will feed the analytics. Nothing else works until the participants are both trusted and instrumented. Just as players are issued limited passes before anything else, the container is built to run as a non-root user from the start. Just as the tracking sensors are fitted before the match so data flows, the /metrics endpoint is built in before monitoring is wired up. Just as a player's credentials are checked privately, the password is read from a private secret file. This shows why foundation comes first: a stack can only be secured and observed if its core components are trustworthy and instrumented from birth.

dockerfile

# api/Dockerfile — hardened, non-root, instrumented
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
RUN adduser --system --no-create-home --uid 10001 scorer
USER scorer                                  # never run as root
EXPOSE 5000
CMD ["python", "app.py"]

# api/app.py — reads secret from file, exposes /healthz and /metrics
import os
from flask import Flask, jsonify
from prometheus_client import Counter, Gauge, generate_latest

app = Flask(__name__)
REQUESTS = Counter('scorecard_requests_total', 'API requests', ['endpoint'])
RUN_RATE = Gauge('scorecard_live_run_rate', 'Current live run rate')

def db_password():
    with open(os.environ['DB_PASSWORD_FILE']) as fh:
        return fh.read().strip()

@app.get('/healthz')
def healthz():
    REQUESTS.labels('/healthz').inc()
    return jsonify(status='healthy'), 200

@app.get('/api/run-rate')
def run_rate():
    REQUESTS.labels('/api/run-rate').inc()
    RUN_RATE.set(8.2)                        # demo value, normally from cache
    return jsonify(live_run_rate=8.2)

@app.get('/metrics')
def metrics():
    return generate_latest(), 200, {'Content-Type': 'text/plain'}

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Step 2 — Core Logic

Step 2 assembles the Compose file with the security controls that define this exercise: two networks isolating the data tier from the edge, per-service hardening flags, a mounted secret for the database password, and health checks with conditional dependencies. Only the proxy publishes a host port; the database and cache live solely on the internal backend network. Each service drops all capabilities, runs with no-new-privileges, and the API and proxy use read-only root filesystems with tmpfs scratch areas. This is where least-privilege moves from a single image into the whole stack's topology.

Analogy🏏Cricket

🏏 Think of it like cricket: with vetted players ready, the organisers now design the venue's security plan — concentric access zones, a locked strongroom for valuables, and a rule that each role carries only the keys it needs. The match cannot be called secure until the whole ground is laid out this way. Just as the strongroom sits in the most restricted zone, the database sits on the isolated backend network. Just as each role carries only its necessary keys, each service is granted only its necessary capabilities. Just as only the main gate faces the public, only the proxy publishes a host port. This shows why core logic is the security plan: individual trust is not enough until the entire layout enforces least privilege between every part.

yaml

# compose.yaml — hardened, segmented stack
services:
  edge:
    image: nginx:1.27
    ports: ["8080:80"]                 # only public entry point
    volumes: ["./proxy/nginx.conf:/etc/nginx/nginx.conf:ro"]
    networks: [frontend]
    read_only: true
    tmpfs: ["/var/cache/nginx", "/var/run"]
    cap_drop: ["ALL"]
    cap_add: ["NET_BIND_SERVICE"]      # only to bind port 80
    security_opt: ["no-new-privileges:true"]
    depends_on:
      scorecard-api: { condition: service_healthy }

  scorecard-api:
    build: ./api
    networks: [frontend, backend]
    environment: { DB_PASSWORD_FILE: /run/secrets/db_password }
    secrets: [db_password]
    read_only: true
    tmpfs: ["/tmp"]
    cap_drop: ["ALL"]
    security_opt: ["no-new-privileges:true"]
    healthcheck:
      test: ["CMD","python","-c","import urllib.request,sys;sys.exit(0 if urllib.request.urlopen('http://localhost:5000/healthz').status==200 else 1)"]
      interval: 5s
      timeout: 3s
      retries: 5
      start_period: 10s
    depends_on:
      match-db: { condition: service_healthy }

  match-db:
    image: postgres:16
    networks: [backend]                # no host port — isolated
    environment:
      POSTGRES_USER: admin
      POSTGRES_DB: cricket
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets: [db_password]
    volumes: ["match-data:/var/lib/postgresql/data"]
    security_opt: ["no-new-privileges:true"]
    healthcheck:
      test: ["CMD-SHELL","pg_isready -U admin -d cricket"]
      interval: 5s
      timeout: 3s
      retries: 5
      start_period: 10s

  score-cache:
    image: redis:7
    networks: [backend]
    security_opt: ["no-new-privileges:true"]

networks: { frontend: {}, backend: {} }
volumes: { match-data: {} }
secrets:
  db_password: { file: ./secrets/db_password.txt }

Step 3 — Integration & Enhancement

Step 3 adds the observability layer and brings everything together. You append Prometheus and Grafana services on the backend network, give Prometheus a scrape config that targets the API's /metrics endpoint by service name, and expose Grafana so you can build a dashboard. Now the hardened, segmented stack is also fully observable: Prometheus pulls the API's request counters and run-rate gauge every fifteen seconds, and Grafana visualises the trends. This integration turns a secure-but-opaque stack into one you can both defend and watch.

Analogy🏏Cricket

🏏 Think of it like cricket: once the venue is secured, the final step before going live is the broadcast and analytics setup — cameras positioned, the stats engine wired to the scoreboard, and the analyst's dashboard switched on. Only then can everyone see and understand the match as it unfolds. Just as the stats engine pulls figures from the scoreboard at a steady cadence, Prometheus scrapes the API's metrics every fifteen seconds. Just as the analyst's dashboard turns raw numbers into readable trends, Grafana turns scraped metrics into graphs. Just as the analytics gear connects only to the official feed, the monitoring services sit on the internal network with the API. This shows why integration completes the stack: security keeps it safe, but observability is what makes it understandable and operable.

yaml

# Append to compose.yaml: the observability services
  prometheus:
    image: prom/prometheus:latest
    networks: [backend]
    volumes:
      - ./observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro
    ports: ["9090:9090"]
    security_opt: ["no-new-privileges:true"]

  grafana:
    image: grafana/grafana:latest
    networks: [backend]
    ports: ["3000:3000"]
    environment: { GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/db_password }
    secrets: [db_password]
    security_opt: ["no-new-privileges:true"]

# observability/prometheus.yml — scrape the API by service name
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: scorecard
    static_configs:
      - targets: ["scorecard-api:5000"]   # resolved by Compose DNS

Step 4 — Testing & Verification

Now verify every property the stack is supposed to guarantee. Bring it up, confirm the API becomes healthy, exercise it through the proxy, prove the database is unreachable from the host, scan the API image for vulnerabilities, and check that Prometheus is successfully scraping the metrics. Each check maps to one of the techniques you applied, so a green run is concrete evidence the hardening and observability actually work rather than merely being declared.

Analogy🏏Cricket

🏏 Think of it like cricket: this verification step is the post-match review that confirms every part of the game went to plan. Just as a captain checks the scorecard to confirm the total was reached, the wickets fell as expected, and every fielder did his job, you confirm nginx is reachable at localhost:8080, that you observed it via logs, inspect and exec, and that you stopped and restarted it cleanly. Just as a rolling substitution is checked to have swapped a fresh player in without stopping play, you verify the restart-policy version took over and the self-cleaning interactive container left no trace. And just as a diligent groundsman clears the pitch and confirms nothing is left behind before locking up, you stop and remove the web container, prune leftover stopped ones, and check with `docker ps -a` that the field is truly empty. The payoff: proof the full lifecycle worked and the host is left spotless.

bash

# Bring up the full stack and verify each guarantee
docker compose up -d --build

# 1) API health and a request through the public proxy:
curl -fsS localhost:8080/api/run-rate        # {"live_run_rate":8.2}

# 2) Isolation: the database must NOT be reachable from the host:
! curl -fsS --max-time 2 localhost:5432 2>/dev/null && echo 'db correctly isolated'

# 3) Hardening: API runs as non-root with no capabilities:
docker compose exec scorecard-api id        # uid=10001(scorer)  -- not root
docker inspect $(docker compose ps -q scorecard-api) \
  --format '{{.HostConfig.ReadonlyRootfs}} {{.HostConfig.CapDrop}}'
# true [ALL]

# 4) Vulnerability scan of the API image (fail on fixable high/critical):
docker scout cves $(docker compose images -q scorecard-api) \
  --only-fixed --only-severity critical,high

# 5) Observability: Prometheus shows the target as UP and metrics flowing:
curl -s 'localhost:9090/api/v1/targets' | grep -o '"health":"up"'
curl -s localhost:8080/api/run-rate >/dev/null   # generate traffic
curl -s 'localhost:9090/api/v1/query?query=scorecard_requests_total'

docker compose down -v

Warning: A read-only root filesystem will crash any service that needs to write to disk unless you provide a writable tmpfs for its scratch paths. Nginx writes to /var/cache/nginx and /var/run, and many apps write to /tmp; if you set read_only: true without the matching tmpfs mounts, the container fails to start with a 'read-only file system' error. Add a tmpfs entry for each path the service must write, and check the logs to find any path you missed.

Extension Challenge: Promote this stack to multi-host resilience and richer observability. First, convert it to a Swarm stack with docker stack deploy, give scorecard-api three replicas behind the proxy on an encrypted overlay network, and confirm self-healing by killing a replica. Second, add a pre-built Grafana dashboard provisioned from a file that graphs scorecard_requests_total and scorecard_live_run_rate. Third, add a centralised logging service and switch the API to structured JSON logs that include a request ID, then correlate a spike in the metrics with the matching log lines.

Network segmentation isolates the data tier: only the edge proxy publishes a host port while the database and cache live on an internal network.
Stack-wide hardening means every service drops all capabilities, sets no-new-privileges, and the API and proxy run read-only with tmpfs scratch space.
Secrets are delivered as mounted files, never environment variables, so the database password never appears in inspect output or logs.
A read-only root filesystem requires a tmpfs for each writable path, or the service fails to start — verification catches this immediately.
Image scanning in the workflow surfaces fixable vulnerabilities before deployment, turning an invisible liability into an actionable list.
An instrumented /metrics endpoint plus a Prometheus scrape config and Grafana make the secured stack observable as well as defensible.