Capstone — Monitor CricketPulse Through a Match-Day Spike

In this capstone project you will deploy the complete CricketPulse observability stack — Prometheus, Alertmanager, Loki, Promtail, OTel Collector, Tempo, and Grafana — instrument a multi-endpoint FastAPI service, run a match-day traffic spike simulation, and use the full three-signal investigation workflow (metric → log → trace) to diagnose two injected faults. This project brings together every skill from all six modules.

Analogy🏏Cricket

🏏 Think of it like cricket: This is the IPL Final for your observability skills. You're setting up the complete broadcast production suite — cameras, tracking systems, archive, and live analysis app — for the biggest match of the season. The match will surface two unexpected incidents. Your job is to find, diagnose, and document them using only the observability signals you've built, just as the match analyst team would diagnose a collapse using ball-by-ball data.

Phase 1 — Deploy the Full Stack

Create a complete docker-compose.yaml that brings up all seven services: FastAPI app, Prometheus, Alertmanager, Loki, Promtail, OTel Collector, Tempo, and Grafana. Use the configurations from each module's exercises. Verify all services are healthy before proceeding to Phase 2.

Analogy🏏Cricket

🏏 Think of it like cricket: Before the first ball is bowled, the broadcast director runs a full systems check: cameras online, microphones live, archive recording, graphics ready. Your Phase 1 is that pre-match systems check — every service healthy, every data flow verified.

yaml

# docker-compose.yaml (complete stack)
version: "3.8"
services:
  # ── Application ─────────────────────────────────────────
  cricketpulse:
    build: ./app
    ports:
      - "8000:8000"
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317
    depends_on: [otel-collector, loki]

  # ── Metrics ─────────────────────────────────────────────
  prometheus:
    image: prom/prometheus:latest
    command:
      - "--config.file=/etc/prometheus/prometheus.yaml"
      - "--enable-feature=exemplar-storage"
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yaml
      - ./alert-rules.yaml:/etc/prometheus/alert-rules.yaml
    ports:
      - "9090:9090"

  alertmanager:
    image: prom/alertmanager:latest
    volumes:
      - ./alertmanager.yaml:/etc/alertmanager/alertmanager.yaml
    ports:
      - "9093:9093"

  # ── Logs ────────────────────────────────────────────────
  loki:
    image: grafana/loki:latest
    command: -config.file=/etc/loki/local-config.yaml
    ports:
      - "3100:3100"

  promtail:
    image: grafana/promtail:latest
    command: -config.file=/etc/promtail/config.yaml
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./promtail.yaml:/etc/promtail/config.yaml

  # ── Traces ──────────────────────────────────────────────
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.100.0
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol/config.yaml
    ports:
      - "4317:4317"
    depends_on: [tempo]

  tempo:
    image: grafana/tempo:latest
    command: -config.file=/etc/tempo.yaml
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
      - tempo-data:/var/tempo
    ports:
      - "3200:3200"

  # ── Visualisation ────────────────────────────────────────
  grafana:
    image: grafana/grafana:latest
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning
    ports:
      - "3000:3000"

volumes:
  tempo-data:

bash

# Phase 1 health check script
#!/bin/bash
set -e

services=(
  "prometheus:9090/-/healthy"
  "alertmanager:9093/-/healthy"
  "loki:3100/ready"
  "tempo:3200/ready"
  "grafana:3000/api/health"
  "cricketpulse:8000/health"
)

for svc in "${services[@]}"; do
  host="${svc%%:*}"
  rest="${svc#*:}"
  port="${rest%%/*}"
  path="${rest#*/}"
  status=$(curl -s -o /dev/null -w "%{http_code}" "http://localhost:${port}/${path}")
  if [ "$status" = "200" ]; then
    echo "PASS $host (HTTP $status)"
  else
    echo "FAIL $host (HTTP $status)"
    exit 1
  fi
done
echo "All services healthy"

Phase 2 — Instrument and Configure Dashboards

Ensure the FastAPI app emits the three signals: Prometheus metrics with exemplars, JSON-structured logs with trace IDs to Loki, and OTLP traces to Tempo. Configure a Grafana dashboard with four rows: Status (Stat panels for error rate, SLO budget remaining), Trends (time-series for request rate, p99 latency, error rate), Logs (Loki panel with derived field for trace correlation), and Traces (Tempo search panel). Add two alert rules: HighErrorRate and SLOBurnRateFast.

Analogy🏏Cricket

🏏 Think of it like cricket: Set up the complete scorer's table: live scoreboard (Status row), over-by-over run rate chart (Trends row), ball-by-ball commentary feed (Logs panel), and the Hawk-Eye replay archive (Tempo panel). The alert rules are the early-warning lights that tell the captain when the match situation is becoming critical.

json

# Grafana dashboard JSON skeleton (save to grafana/provisioning/dashboards/)
{
  "title": "CricketPulse Match Day",
  "refresh": "30s",
  "panels": [
    {
      "title": "Error Rate",
      "type": "stat",
      "gridPos": {"x":0,"y":0,"w":6,"h":4},
      "targets": [{"expr": "rate(http_requests_total{status=~'5..'}[5m]) / rate(http_requests_total[5m]) * 100", "legendFormat": "Error %"}],
      "fieldConfig": {"defaults": {"thresholds": {"steps": [{"value": 0, "color": "green"}, {"value": 1, "color": "yellow"}, {"value": 5, "color": "red"}]}}}
    },
    {
      "title": "Error Budget Remaining",
      "type": "stat",
      "gridPos": {"x":6,"y":0,"w":6,"h":4},
      "targets": [{"expr": "slo:error_budget_remaining:ratio * 100", "legendFormat": "Budget %"}]
    }
  ]
}

Phase 3 — Run the Match-Day Simulation

Run the provided traffic simulation script that generates a realistic match-day load profile: steady baseline traffic, a spike when the match starts (12x baseline), a fault injection at the 20-minute mark (elevated error rate on the score endpoint), and a second fault at the 35-minute mark (latency injection on the checkout endpoint). Your task is to detect both faults using the observability stack and document the investigation path for each.

Analogy🏏Cricket

🏏 Think of it like cricket: The match begins. Rohit Sharma and Virat Kohli open to steady applause (baseline traffic). A six off the first ball sends the crowd wild — 12x the normal activity (traffic spike). At over 5, Rohit takes an unexpected blow (fault 1: error rate spike). At over 10, the scoring system slows as the DRS review queue jams (fault 2: latency injection). Your job is the analyst: detect both problems using the live feeds before the captain has to make a decision.

python

#!/usr/bin/env python3
# traffic_simulation.py — match-day spike simulator
import asyncio, random, time, httpx, sys

BASE_URL = "http://localhost:8000"
ENDPOINTS = [
    ("GET", "/match/IPL-2025-F1/score", {}),
    ("GET", "/match/IPL-2025-F1/commentary", {}),
    ("POST", "/api/checkout", {"order_id": "merch-001"}),
    ("GET", "/health", {}),
]

async def make_request(client, method, path, body):
    try:
        if method == "GET":
            r = await client.get(f"{BASE_URL}{path}", timeout=5)
        else:
            r = await client.post(f"{BASE_URL}{path}", json=body, timeout=5)
        return r.status_code
    except Exception:
        return 0

async def load_phase(name, rps, duration_s, error_inject=False, latency_inject=False):
    print(f"Phase: {name} | {rps} RPS | {duration_s}s")
    async with httpx.AsyncClient() as client:
        end = time.time() + duration_s
        while time.time() < end:
            tasks = []
            for _ in range(rps):
                method, path, body = random.choice(ENDPOINTS)
                if error_inject and "/score" in path:
                    path = "/score/invalid-id-to-trigger-error"
                tasks.append(make_request(client, method, path, body))
            await asyncio.gather(*tasks)
            await asyncio.sleep(1)

async def main():
    await load_phase("Baseline",      rps=5,  duration_s=600)
    await load_phase("Match Start",   rps=60, duration_s=600)
    await load_phase("Fault 1: Errors", rps=60, duration_s=300, error_inject=True)
    await load_phase("Recovery",      rps=60, duration_s=300)
    await load_phase("Fault 2: Slow", rps=60, duration_s=300, latency_inject=True)
    await load_phase("Wind Down",     rps=10, duration_s=300)

asyncio.run(main())

Rubric

Your capstone submission should include: a running docker-compose stack (100% health check pass), the completed Grafana dashboard export JSON, two investigation reports (one per fault), and a retrospective noting any alerts that fired unexpectedly or did not fire when expected. Each of the four rubric areas is worth 25 points.

Analogy🏏Cricket

🏏 Think of it like cricket: The capstone rubric is the match-referee's official report card, scoring your performance across four equally weighted areas of 25 points each, just as a player's match rating splits evenly across batting, bowling, fielding, and game awareness rather than rewarding one heroic moment. A running docker-compose stack at 100% health-check pass is your fitness certificate — every system green before play. The Grafana dashboard export is your prepared game plan, submitted and reviewable. The two investigation reports, one per injected fault, are your two dismissals earned — proof you actually read the signals and traced each failure to its cause under match-day pressure, not just watched the score. And the retrospective on alerts that fired unexpectedly or stayed silent is your honest self-review, like a captain admitting which field placements misfired. Just as a balanced rating rewards the complete cricketer, the rubric rewards the complete operator: stack, dashboard, diagnosis, and reflection. The payoff: being judged across all four proves you can run observability end-to-end through a real traffic spike — the whole course made concrete.

bash

# Rubric: 100 points total

# 1. Infrastructure (25 pts)
#    - All 7 services pass health check (10)
#    - Prometheus scraping app metrics (5)
#    - Loki receiving structured logs with trace_id (5)
#    - Tempo receiving traces from OTel Collector (5)

# 2. Observability Coverage (25 pts)
#    - Grafana dashboard with all 4 rows present (10)
#    - Exemplars visible on time-series panels (5)
#    - Loki derived field creates clickable trace link (5)
#    - Alert rules defined and validated with promtool (5)

# 3. Incident Investigation (25 pts per fault)
#    Each fault report must include:
#    - Screenshot of metric panel showing the anomaly
#    - Log lines from Loki correlated to the fault window
#    - Trace waterfall screenshot from Tempo
#    - Root cause statement (which endpoint / service / cause)
#    - Time to detect from fault injection to diagnosis (target < 5 min)

# 4. SLO and Alerting (25 pts)
#    - Error budget burn-down chart shows fault 1 consuming budget
#    - SLOBurnRateFast alert fired during fault 1 (check Alertmanager)
#    - Post-simulation error budget remaining > 60%
#    - Retrospective identifies one alert improvement

If Tempo shows no traces after the simulation, check that the OTel Collector exporter endpoint matches the Tempo receiver address (tempo:4317 for gRPC). Run 'docker compose logs otel-collector' to see if spans are being forwarded successfully before debugging the application SDK.

For the retrospective, a common finding is that fault 2 (latency injection) triggers no alert if the p99 threshold was set too loosely. If that happens, use it as a real finding: propose a tighter SLO latency target and calculate the error budget impact before committing to it. This is exactly the kind of alert calibration work that happens after real incidents.

The full stack runs as eight Docker Compose services with health checks before any simulation traffic.
All three signals must be connected: Prometheus exemplars → Loki derived fields → Tempo waterfalls.
Fault detection target is under 5 minutes from injection using the dashboard and Explore view.
Error budget burn-down provides the business context for why both faults matter beyond raw metrics.
The retrospective is as important as the investigation — it drives the next round of alert calibration.

Capstone — Monitor CricketPulse Through a Match-Day Spike

Phase 1 — Deploy the Full Stack

Phase 2 — Instrument and Configure Dashboards

Phase 3 — Run the Match-Day Simulation

Rubric

Submit your capstone project