What You'll Build
In this exercise you will build and operate a complete four-service IPL analytics platform using Docker Compose, applying every technique from M2's reading lessons in a single integrated workflow. The platform consists of a FastAPI scorecard service, a PostgreSQL database with persistent storage, a Redis cache for request rate counting, and an Nginx reverse proxy — wired together with user-defined networks for micro-segmentation, named volumes for durable state, healthchecks with `condition: service_healthy` dependency ordering, and a development profile that activates pgAdmin without modifying the base Compose file. You will then build the API image with BuildKit, scan it with Trivy, tag it with the Git commit SHA, and push it to a local Docker registry — completing the pipeline from local development environment to registry-ready artefact in a single Compose-managed workflow.
The deliberate design of this exercise is that every configuration decision in the Compose file should be traceable to a specific principle from the reading lessons: the `condition: service_healthy` dependency is from Lesson 8, the `profiles: [dev]` on pgAdmin is from Lesson 9, the user-defined network subnets are from M1 Lesson 5, and the named volume with OverlayFS bypass is from M1 Lesson 4. Building the platform and verifying each property independently — health dependency ordering, network micro-segmentation, profile activation, volume persistence — proves that each principle works not in isolation but in combination, which is the only form in which they appear in production systems.
Prerequisites
- The `ipl-scorecard` FastAPI project from M1 Exercise (Lesson 6) with a working Dockerfile and passing pytest suite — this exercise extends that project rather than creating a new one.
- Docker Engine v24+ with BuildKit enabled — verify with `docker buildx version` and confirm at least one builder instance is available.
- Trivy v0.50+ installed locally for the scan step — `trivy --version` should return the installed version before beginning.
- Git repository initialised in the project directory — the image tag will be derived from the Git commit SHA using `git rev-parse --short HEAD`.
- A local registry running on port 5001 — `docker run -d -p 5001:5000 --name local_registry registry:2` creates a local OCI-compliant registry suitable for push testing without cloud credentials.
Setup & Project Structure
Extend the M1 exercise project with the additional services, configuration files, and secrets directory that the full platform requires. Keep the Dockerfile from M1 unchanged — this exercise is about the Compose configuration, not the image build. Create the secrets files locally and add them to `.gitignore` immediately, before writing a single line of Compose configuration, to establish the discipline that secrets are never committed to version control.
# Extend the M1 exercise project structure
cd ipl_scorecard_api # from M1 exercise
# ── Create additional configuration files ─────────────────────────────────
mkdir -p nginx secrets
# Nginx reverse proxy configuration
cat > nginx/ipl.conf << 'EOF'
upstream ipl_api {
server ipl_api:8000; # resolves via Docker's embedded DNS on ipl_frontend network
}
server {
listen 80;
location /api/ {
proxy_pass http://ipl_api/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
location /health {
proxy_pass http://ipl_api/health;
}
}
EOF
# ── Create secrets directory and populate with development values ──────────
# Add secrets/ to .gitignore BEFORE creating any files inside it
echo "secrets/" >> .gitignore
echo ".env" >> .gitignore
# Development-only secrets (never committed)
echo "ipl_dev_postgres_password" > secrets/pg_password.txt
echo "ipl_dev_pgadmin_password" > secrets/pgadmin_password.txt
# ── Update main.py to add Redis request counting ──────────────────────────
cat >> main.py << 'EOF'
import os
import redis
# Redis client — URL injected via environment variable
_redis_client = None
def get_redis():
global _redis_client
if _redis_client is None:
redis_url = os.environ.get("REDIS_URL", "redis://localhost:6379/0")
_redis_client = redis.from_url(redis_url, decode_responses=True)
return _redis_client
@app.get("/stats")
async def request_stats() -> dict:
r = get_redis()
# Increment a counter per player request for analytics
keys = r.keys("requests:player:*")
stats = {k.split(":")[-1]: int(r.get(k) or 0) for k in keys}
return {"player_request_counts": stats, "total": sum(stats.values())}
EOF
# Add redis to requirements.txt
echo "redis==5.0.1" >> requirements.txt
# Start a local registry for push testing
docker run -d --name local_registry -p 5001:5000 -v /tmp/local_registry_data:/var/lib/registry registry:2
echo "Setup complete. Local registry available at localhost:5001"
docker ps --filter name=local_registryStep 1 — Foundation
Write the complete `compose.yml` with all four services, explicit subnet declarations, named volumes, healthchecks, and the development profile for pgAdmin. Write the `compose.override.yml` with the development bind mount for source code hot-reload. Write the `.env` file with variable-substituted values. Run `docker compose config` to verify the merged configuration before starting any services — this validation step catches merge errors, missing variable references, and syntax mistakes before they manifest as confusing runtime failures.
# compose.yml — complete four-service IPL analytics platform
version: "3.9"
services:
ipl_postgres:
image: postgres:16-alpine
environment:
POSTGRES_DB: ${DB_NAME:-cricket_stats}
POSTGRES_USER: ${DB_USER:-rohit_admin}
POSTGRES_PASSWORD_FILE: /run/secrets/pg_password
volumes:
- ipl_match_data:/var/lib/postgresql/data
networks: [ipl_backend]
secrets: [pg_password]
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-rohit_admin} -d ${DB_NAME:-cricket_stats}"]
interval: 10s
timeout: 5s
retries: 5
start_period: 30s
restart: on-failure
ipl_redis:
image: redis:7-alpine
command: redis-server --appendonly yes --appendfsync everysec --maxmemory 64mb --maxmemory-policy allkeys-lru
volumes:
- ipl_redis_aof:/data
networks: [ipl_backend]
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 3
restart: on-failure
ipl_api:
image: ${REGISTRY:-localhost:5001}/ipl-scorecard:${IMAGE_TAG:-dev}
build:
context: .
dockerfile: Dockerfile
target: ipl_runtime
environment:
DATABASE_URL: "postgresql://${DB_USER:-rohit_admin}@ipl_postgres:5432/${DB_NAME:-cricket_stats}"
REDIS_URL: "redis://ipl_redis:6379/0"
networks: [ipl_frontend, ipl_backend]
depends_on:
ipl_postgres: {condition: service_healthy}
ipl_redis: {condition: service_healthy}
healthcheck:
test: ["CMD", "python3", "-c",
"import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
interval: 15s
timeout: 3s
retries: 3
start_period: 10s
mem_limit: 256m
cpus: "0.5"
read_only: true
tmpfs: [/tmp:size=32m,mode=1777]
cap_drop: ["ALL"]
security_opt: ["no-new-privileges:true"]
user: "10001:10001"
restart: on-failure
ipl_nginx:
image: nginx:alpine
ports: ["127.0.0.1:80:80"]
volumes:
- type: bind
source: ./nginx/ipl.conf
target: /etc/nginx/conf.d/default.conf
read_only: true
networks: [ipl_frontend]
depends_on:
ipl_api: {condition: service_healthy}
restart: on-failure
# pgAdmin: dev profile only — never starts in CI or production
ipl_pgadmin:
profiles: [dev]
image: dpage/pgadmin4:latest
environment:
PGADMIN_DEFAULT_EMAIL: rohit@ipl.example.com
PGADMIN_DEFAULT_PASSWORD_FILE: /run/secrets/pgadmin_password
ports: ["127.0.0.1:5050:80"]
networks: [ipl_backend]
secrets: [pgadmin_password]
networks:
ipl_frontend: {driver: bridge, ipam: {config: [{subnet: 192.168.100.0/24}]}}
ipl_backend: {driver: bridge, ipam: {config: [{subnet: 192.168.101.0/24}]}}
volumes:
ipl_match_data: {}
ipl_redis_aof: {}
secrets:
pg_password: {file: ./secrets/pg_password.txt}
pgadmin_password: {file: ./secrets/pgadmin_password.txt}
# compose.override.yml — development hot-reload (auto-loaded in dev)
# version: "3.9"
# services:
# ipl_api:
# build: {context: ., dockerfile: Dockerfile, target: ipl_runtime}
# volumes:
# - type: bind
# source: ./scorecard_api
# target: /app/scorecard_api
# command: uvicorn main:app --reload --host 0.0.0.0 --port 8000
# environment:
# LOG_LEVEL: DEBUGStep 2 — Core Logic
Build the API image, scan it with Trivy, tag it with the Git commit SHA, and push it to the local registry. Update the `.env` file with the correct `IMAGE_TAG` so that `docker compose up` uses the freshly built and scanned image rather than the `:dev` fallback. This build-scan-tag-push sequence is the local equivalent of the CI pipeline from M1 Lesson 7, demonstrating that the same workflow that runs in GitHub Actions also works locally without any cloud infrastructure.
# Build, scan, tag, push, and update IMAGE_TAG for compose
# ── 1. Derive image tag from Git commit SHA ────────────────────────────────
GIT_SHA=$(git rev-parse --short HEAD)
IMAGE_TAG="sha-$GIT_SHA"
REGISTRY="localhost:5001"
IMAGE="$REGISTRY/ipl-scorecard:$IMAGE_TAG"
echo "Building: $IMAGE"
# ── 2. Build with BuildKit (remote cache not applicable for local registry) ──
docker buildx build --tag $IMAGE --tag $REGISTRY/ipl-scorecard:latest --load \ # load into Docker daemon (required for local use)
.
# ── 3. Scan with Trivy — hard gate ─────────────────────────────────────────
trivy image --exit-code 1 --severity CRITICAL,HIGH --format table $IMAGE
# If scan passes, update .env so compose uses the scanned image
# ── 4. Update IMAGE_TAG in .env ────────────────────────────────────────────
if grep -q "^IMAGE_TAG=" .env; then
# Update existing entry
sed -i "s/^IMAGE_TAG=.*/IMAGE_TAG=$IMAGE_TAG/" .env
else
# Append new entry
echo "IMAGE_TAG=$IMAGE_TAG" >> .env
echo "REGISTRY=localhost:5001" >> .env
fi
echo "IMAGE_TAG=$IMAGE_TAG" >> .env
# ── 5. Push to local registry ─────────────────────────────────────────────
docker push $IMAGE
docker push $REGISTRY/ipl-scorecard:latest
echo "Pushed: $IMAGE"
docker images "$REGISTRY/ipl-scorecard" --format "table {{.Tag}} {{.Size}} {{.CreatedAt}}" Step 3 — Integration & Enhancement
Start the full platform and verify each property in sequence: health-dependency ordering by watching `docker compose ps` until all services reach `healthy`, network micro-segmentation by confirming Nginx cannot reach PostgreSQL, volume persistence by inserting a row into PostgreSQL then recreating all containers and verifying the row survives, and profile activation by confirming pgAdmin is absent from the standard startup and present after `--profile dev` activation. Each verification produces observable evidence — a `docker exec ping` result, a `psql SELECT` output, a `docker compose ps` status line — that proves the configuration is correct rather than merely present.
# Verification sequence: each test confirms one configuration property
# ── Start platform and wait for all services to reach healthy ──────────────
docker compose up -d
echo "Waiting for all services to reach 'healthy' status..."
until [ "$(docker compose ps --format json | python3 -c "
import sys, json
services = [json.loads(l) for l in sys.stdin if l.strip()]
all_healthy = all(s.get('Health') in ('healthy', '') for s in services
if s.get('Service') not in ('ipl_pgadmin',))
print('yes' if all_healthy else 'no')
")" = "yes" ]; do
docker compose ps --format "table {{.Service}} {{.Status}} {{.Health}}"
sleep 5
done
echo "All services healthy ✓"
# ── Verify health-dependency ordering ─────────────────────────────────────
# Check startup times: api must start AFTER postgres and redis are healthy
docker inspect ipl_postgres --format "{{.State.StartedAt}}"
docker inspect ipl_redis --format "{{.State.StartedAt}}"
docker inspect ipl_api --format "{{.State.StartedAt}}"
# api StartedAt must be LATER than both postgres and redis StartedAt ✓
# ── Verify network micro-segmentation ─────────────────────────────────────
# Nginx is on ipl_frontend only — must NOT reach postgres (ipl_backend only)
docker exec ipl_nginx ping -c 1 ipl_postgres 2>&1 | grep -E "unknown host|Name or service"
# Expected: "ping: ipl_postgres: Name or service not known" ✓
# API bridges both networks — must reach postgres AND nginx
docker exec ipl_scorecard_api_1 ping -c1 ipl_postgres 2>&1 | grep "1 packets transmitted"
# Expected: 1 packets transmitted, 1 received ✓
# ── Verify volume persistence (data survives container recreation) ─────────
# Insert test data into PostgreSQL
docker exec ipl_postgres psql -U rohit_admin -d cricket_stats -c "
CREATE TABLE IF NOT EXISTS ipl_test (message TEXT, inserted_at TIMESTAMPTZ DEFAULT NOW());
INSERT INTO ipl_test(message) VALUES ('Rohit Sharma: 264 not out');
"
# Recreate the postgres container (does NOT remove named volumes)
docker compose stop ipl_postgres
docker compose rm -f ipl_postgres
docker compose up -d ipl_postgres
# Wait for postgres to be healthy again
until docker inspect ipl_postgres --format "{{.State.Health.Status}}" | grep -q "healthy"; do sleep 3; done
# Query the test row — must survive container recreation
docker exec ipl_postgres psql -U rohit_admin -d cricket_stats -c "SELECT message FROM ipl_test;"
# Expected: Rohit Sharma: 264 not out — DATA SURVIVED ✓
# ── Verify profile activation ─────────────────────────────────────────────
# pgAdmin must NOT be running without the dev profile
docker compose ps --format json | python3 -c "
import sys, json
for l in sys.stdin:
if l.strip():
s = json.loads(l)
if 'pgadmin' in s.get('Service', '').lower():
print('FAIL: pgAdmin running without dev profile')
exit(1)
print('PASS: pgAdmin not running without dev profile ✓')
"
# Activate dev profile — pgAdmin should start
docker compose --profile dev up -d ipl_pgadmin
docker compose ps --filter service=ipl_pgadmin
# Expected: ipl_pgadmin Up — http://localhost:5050 ✓
# ── API functional test ────────────────────────────────────────────────────
curl -s http://localhost/api/batters/rohit_sharma | python3 -m json.tool
# Expected: {"name": "Rohit Sharma", "runs": 3853, ...}
curl -s http://localhost/api/health
# Expected: {"status": "healthy", "service": "ipl-scorecard"}
echo "All verification steps passed ✓" Step 4 — Testing & Verification
Run the complete verification checklist in sequence. Each item in the checklist produces a binary pass or fail observable output — a ping failure, a health status string, a SQL query result — rather than a subjective assessment. The checklist is the operational equivalent of the ICC referee's post-match certification: each item is checked and signed off independently, and the platform is considered correctly configured only when every item passes. Any failure points to the specific configuration element that requires correction.
# Final checklist: run all verifications and confirm pass/fail for each
echo "=== IPL Analytics Platform — M2 Exercise Verification Checklist ==="
# 1. All core services healthy (excluding dev-only pgadmin)
echo -n "1. All core services healthy: "
UNHEALTHY=$(docker compose ps --format json | python3 -c "
import sys, json
for l in sys.stdin:
if l.strip():
s = json.loads(l)
if 'pgadmin' not in s.get('Service','') and s.get('Health') not in ('healthy',''):
print(s['Service'])
" 2>/dev/null)
[ -z "$UNHEALTHY" ] && echo "PASS ✓" || echo "FAIL ✗ — unhealthy: $UNHEALTHY"
# 2. API started after both postgres and redis (dependency ordering works)
echo -n "2. Dependency ordering enforced: "
PG_START=$(docker inspect ipl_postgres --format "{{.State.StartedAt}}" 2>/dev/null)
API_START=$(docker inspect ipl_scorecard_api_1 --format "{{.State.StartedAt}}" 2>/dev/null)
[ "$API_START" \> "$PG_START" ] && echo "PASS ✓" || echo "FAIL ✗"
# 3. Nginx cannot resolve postgres (network micro-segmentation)
echo -n "3. Network micro-segmentation: "
docker exec ipl_nginx ping -c1 ipl_postgres > /dev/null 2>&1 && echo "FAIL ✗" || echo "PASS ✓"
# 4. Data persists in named volume after container recreation
echo -n "4. Named volume persistence: "
ROWS=$(docker exec ipl_postgres psql -U rohit_admin -d cricket_stats -t -c "SELECT COUNT(*) FROM ipl_test;" 2>/dev/null | tr -d ' ')
[ "$ROWS" -gt "0" ] 2>/dev/null && echo "PASS ✓ ($ROWS rows)" || echo "FAIL ✗"
# 5. pgAdmin absent without dev profile
echo -n "5. Profiles: pgAdmin not started: "
docker compose ps --format json 2>/dev/null | grep -q "pgadmin" && echo "FAIL ✗" || echo "PASS ✓"
# 6. API routes correctly through Nginx
echo -n "6. Nginx reverse proxy working: "
STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost/api/health)
[ "$STATUS" = "200" ] && echo "PASS ✓" || echo "FAIL ✗ (HTTP $STATUS)"
# 7. Trivy scan clean (re-run against deployed image)
echo -n "7. Trivy scan: no CRITICAL/HIGH: "
trivy image --severity CRITICAL,HIGH --exit-code 1 --quiet "localhost:5001/ipl-scorecard:latest" > /dev/null 2>&1 && echo "PASS ✓" || echo "FAIL ✗"
# 8. Image in local registry with SHA tag
echo -n "8. Image tagged and pushed to registry: "
docker pull "localhost:5001/ipl-scorecard:sha-$(git rev-parse --short HEAD)" > /dev/null 2>&1 && echo "PASS ✓" || echo "FAIL ✗"
echo "=== Checklist complete ===" Warning: When running `docker compose up -d` after changing `IMAGE_TAG` in `.env`, Compose detects the configuration change but does not automatically pull or rebuild the image. Always run `docker compose build ipl_api` followed by `docker compose up -d --no-deps ipl_api` after updating `IMAGE_TAG` to ensure the new image is used. Running only `docker compose up -d` after a `.env` change will recreate the container with the new environment variables but using whichever image was already pulled, which may still be the old tag if the new tag has not been built yet. The `--no-deps` flag prevents Compose from unnecessarily restarting PostgreSQL and Redis when only the API image changes.
Extension Challenge: Add a Prometheus and Grafana monitoring stack using the `monitoring` profile, connected to the `ipl_backend` network so Prometheus can scrape the FastAPI `/metrics` endpoint. Add `prometheus-fastapi-instrumentator` to `requirements.txt` and mount a `prometheus.yml` configuration as a bind-mount into the Prometheus container. Create a Grafana dashboard JSON that shows request rate, latency P95, and error rate for the `/batters/` endpoints. This extension exercises the profiles pattern for a third service type — observability tooling — and demonstrates how the multi-network topology allows monitoring access to internal services without exposing them through Nginx.
- The `condition: service_healthy` chain — postgres healthy → redis healthy → API starts → API healthy → Nginx starts — is the correct startup sequence that prevents connection errors during initialisation.
- Network micro-segmentation enforced by placing database and cache services on a backend-only network prevents frontend services from establishing direct database connections, even if misconfigured.
- Volume persistence is verified by inserting data, recreating the container, and querying the data — the only way to confirm that named volumes survive container lifecycle events as expected.
- Profiles separate optional services from core services in version-controlled form — pgAdmin is always defined but never started unless explicitly activated, with zero modification to the base Compose file.
- Building with BuildKit, scanning with Trivy before push, tagging with a Git commit SHA, and updating `.env` to reference the scanned tag is the local equivalent of the CI pipeline pattern from Lesson 7.
- Run `docker compose config` before `docker compose up` whenever changing the Compose file or override files to verify the fully merged configuration and catch missing network references or undefined variables early.