What You'll Build
In this exercise you will add production-grade Prometheus instrumentation to a FastAPI service simulating the CricketPulse scoring backend. By the end you will have a running service exposing custom metrics, a Prometheus instance scraping it, and verified PromQL queries returning meaningful data. This is the foundation every subsequent module builds on — without working application metrics, Grafana dashboards, alerting rules, and SLO calculations have nothing to visualise.
Analogy🏏Cricket
🏏 Think of it like cricket: This exercise is like being handed a cricket ground that has no scoreboards, no Hawk-Eye cameras, and no stump microphones, and being told to instrument it before the match starts in 45 minutes. You will install the run-rate display (Prometheus metrics), set up the ball-by-ball commentary feed (structured logs), and mount the ball-tracking sensors (traces) — all in time for Rohit Sharma to face the first delivery. Just as a ground with no instrumentation cannot produce DRS reviews or post-match analytics, a service with no observability cannot produce root cause analysis. The insight is that this exercise simulates exactly the pressure of instrumenting a service before go-live, when adding telemetry is fastest and cheapest.
Prerequisites
- Python 3.10+ and Docker Desktop installed
- Lesson 5 (Prometheus Architecture) and Lesson 7 (Exporters and Custom Metrics) completed
- Basic FastAPI familiarity — you should understand routes and middleware
- Lessons 5 and 6 PromQL concepts — you will write queries to verify your metrics
Setup and Project Structure
Create the project directory, virtual environment, and install dependencies. The project separates metric definitions (metrics.py) from application logic (main.py) to keep instrumentation concerns isolated and testable independently.
Analogy🏏Cricket
🏏 Think of it like cricket: Before a training session you separate the equipment shed from the nets — bats, balls, and cones live in one place, the drills happen in another. Here you do the same: metric definitions live in `metrics.py`, the application logic lives in `main.py`. Just as keeping the equipment shed tidy and separate lets a coach swap in a new drill without rummaging through gear, keeping metric instruments isolated lets you test and reason about instrumentation independently of the code it measures. Just as every net session reuses the same calibrated stumps and cones rather than improvising new ones each time, every request handler imports the same shared Counter and Histogram rather than redefining them, so aggregation stays consistent. Just as a well-organised setup means the players walk straight in and start practising, a clean venv and clear module split means you can start emitting and querying metrics immediately. The payoff: isolating instrumentation concerns up front makes the metrics testable, reusable, and hard to accidentally break when the business logic changes.
bash
mkdir cricketpulse-metrics && cd cricketpulse-metrics
python3 -m venv venv && source venv/bin/activate
pip install fastapi uvicorn prometheus_client httpx
# Project structure
cricketpulse-metrics/
app/
main.py
metrics.py
prometheus/
prometheus.yml
docker-compose.yml
Step 1 — Define Metrics
Create metrics.py with all four metric types. Centralising metric definitions in one module prevents duplicate registration errors (prometheus_client raises ValueError if the same metric name is registered twice) and makes it easy to audit your instrumentation surface.
Analogy🏏Cricket
🏏 Think of it like cricket: Centralising metrics in metrics.py is like the ICC's centralised statistics bureau — one authoritative source for all official records, rather than each team's analyst maintaining their own incompatible spreadsheet. When Virat Kohli's career average is updated, it updates in one place and propagates everywhere. When you add a new metric label, you change one file and all importers benefit. The insight is that instrumentation sprawl — metrics defined across dozens of files — creates the same reconciliation problems that multiple independent scorecards would create.
python
# app/metrics.py — centralise all metric definitions
from prometheus_client import Counter, Gauge, Histogram, REGISTRY
from prometheus_client import CollectorRegistry
# Request tracking
HTTP_REQUESTS = Counter(
'cricketpulse_http_requests_total',
'Total HTTP requests received',
['endpoint', 'method', 'status_code']
)
# Active connections (WebSocket simulation)
ACTIVE_CONNECTIONS = Gauge(
'cricketpulse_active_connections',
'Number of active connections'
)
# Latency distribution
REQUEST_LATENCY = Histogram(
'cricketpulse_http_request_duration_seconds',
'HTTP request latency',
['endpoint'],
buckets=[0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0]
)
# Business metric: score cache performance
CACHE_HITS = Counter(
'cricketpulse_cache_hits_total',
'Cache hit count',
['cache_type']
)
CACHE_MISSES = Counter(
'cricketpulse_cache_misses_total',
'Cache miss count',
['cache_type']
)
Step 2 — Instrument the FastAPI App
Add a middleware that wraps every request with metric recording, and mount the /metrics endpoint using make_asgi_app(). The middleware approach ensures every route is automatically instrumented without per-route boilerplate.
Analogy🏏Cricket
🏏 Think of it like cricket: The middleware approach mirrors the third umpire's role in DRS — a system-level observer that automatically reviews every decision without the on-field umpires needing to actively invoke it for each ball. Every HTTP request passes through the middleware just as every DRS-eligible decision passes through the third umpire review system. The insight is that middleware-level instrumentation is preferable to per-handler instrumentation because it cannot be accidentally omitted from a new route.
python
# app/main.py — FastAPI with Prometheus instrumentation
from fastapi import FastAPI, Request, Response
from prometheus_client import make_asgi_app
import time, random, asyncio
from app.metrics import (
HTTP_REQUESTS, ACTIVE_CONNECTIONS, REQUEST_LATENCY,
CACHE_HITS, CACHE_MISSES
)
app = FastAPI(title='CricketPulse Scoring Service')
# Mount /metrics endpoint
metrics_app = make_asgi_app()
app.mount('/metrics', metrics_app)
@app.middleware('http')
async def metrics_middleware(request: Request, call_next):
start = time.time()
ACTIVE_CONNECTIONS.inc()
try:
response = await call_next(request)
status = str(response.status_code)
HTTP_REQUESTS.labels(
endpoint=request.url.path,
method=request.method,
status_code=status
).inc()
REQUEST_LATENCY.labels(endpoint=request.url.path).observe(
time.time() - start
)
return response
finally:
ACTIVE_CONNECTIONS.dec()
@app.get('/live_score')
async def live_score():
# Simulate cache hit/miss logic
if random.random() < 0.8:
CACHE_HITS.labels(cache_type='redis').inc()
return {'score': '245/4', 'overs': '38.2', 'source': 'cache'}
else:
CACHE_MISSES.labels(cache_type='redis').inc()
await asyncio.sleep(random.uniform(0.05, 0.2))
return {'score': '245/4', 'overs': '38.2', 'source': 'upstream'}
@app.get('/scorecard')
async def scorecard():
await asyncio.sleep(random.uniform(0.1, 0.5)) # DB query simulation
return {'batters': [], 'bowlers': []}
@app.get('/health')
async def health():
return {'status': 'ok'}
Step 3 — Configure Prometheus Scrape
Configure Prometheus to scrape the CricketPulse service and run both services with docker-compose. The web.enable-lifecycle flag allows reloading Prometheus configuration without a full restart, which is essential when iterating on scrape_configs during development.
Analogy🏏Cricket
🏏 Think of it like cricket: The Prometheus scrape configuration is like the ICC assigning an official scorer to each match — a formal declaration that this match will be officially recorded and its statistics will count toward career records. Without the scrape_config entry, the service's /metrics endpoint exists but no one is reading it, just as a match played without an official scorer produces no official statistics. The insight is that metrics emission and metrics collection are separate concerns — a bug in either makes the entire pipeline produce no data.
yaml
# prometheus/prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'cricketpulse'
static_configs:
- targets: ['cricketpulse:8000']
metrics_path: '/metrics'
scrape_interval: 15s
# docker-compose.yml
version: '3.8'
services:
cricketpulse:
build: .
ports:
- '8000:8000'
command: uvicorn app.main:app --host 0.0.0.0 --port 8000
prometheus:
image: prom/prometheus:latest
ports:
- '9090:9090'
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.enable-lifecycle'
Step 4 — Generate Traffic and Verify Queries
Run the traffic generator to produce enough data for meaningful PromQL results, then verify each metric with the Prometheus HTTP API. The 5-minute rate window in PromQL requires at least 5 minutes of data — run the traffic generator for 5 minutes or run it repeatedly.
Analogy🏏Cricket
🏏 Think of it like cricket: A PromQL rate() over a 5-minute window needs at least five minutes of samples, exactly the way you cannot judge a bowler's economy rate from a single delivery — you need a full spell in the book before the average means anything. So you run the traffic generator long enough to fill the window, like bowling several overs so the run-rate calculation has real balls to divide by. Just as you would fire a controlled sequence of deliveries and then check the scorecard to confirm each dismissal and run was recorded correctly, you drive requests and then verify each metric through the Prometheus HTTP API. Just as an economy rate computed over two balls swings wildly and lies, a rate() computed over thirty seconds of data is noisy and misleading, so you honour the full window. The payoff: generating enough traffic before querying is what turns raw counters into trustworthy rates — the practice teaches you that metrics need a warm-up spell before they tell the truth.
python
# generate_traffic.py — simulate match-day load
import httpx, time, random
BASE = 'http://localhost:8000'
ENDPOINTS = ['/live_score', '/live_score', '/live_score', # 3x weight
'/scorecard', '/health']
def burst(n=100, rps=20):
print(f'Sending {n} requests at ~{rps} rps...')
interval = 1.0 / rps
for i in range(n):
ep = random.choice(ENDPOINTS)
try:
r = httpx.get(f'{BASE}{ep}', timeout=2.0)
print(f'{ep} -> {r.status_code}')
except Exception as e:
print(f'{ep} -> ERROR: {e}')
time.sleep(interval)
if __name__ == '__main__':
burst(200, rps=10)
print('Traffic complete. Check Prometheus at http://localhost:9090')
bash
# 1. Start the stack
docker compose up -d
# 2. Verify the /metrics endpoint is live
curl http://localhost:8000/metrics | grep cricketpulse
# 3. Generate traffic
python3 generate_traffic.py
# 4. Query via Prometheus HTTP API
# Request rate per endpoint
curl -s 'http://localhost:9090/api/v1/query?query=sum(rate(cricketpulse_http_requests_total[5m]))+by+(endpoint)' | python3 -m json.tool
# p99 latency
curl -s 'http://localhost:9090/api/v1/query?query=histogram_quantile(0.99,sum(rate(cricketpulse_http_request_duration_seconds_bucket[5m]))+by+(le,endpoint))' | python3 -m json.tool
# Cache hit ratio
curl -s 'http://localhost:9090/api/v1/query?query=rate(cricketpulse_cache_hits_total[5m])/(rate(cricketpulse_cache_hits_total[5m])+rate(cricketpulse_cache_misses_total[5m]))' | python3 -m json.tool
# 5. Open Prometheus UI: http://localhost:9090
# Navigate to Status -> Targets — cricketpulse should show State=UP
Warning: The make_asgi_app() /metrics endpoint is not authenticated by default. In production, place Prometheus behind a VPN or network policy so only authorised Prometheus instances can scrape it. Exposing /metrics publicly leaks service topology, label values, and potentially business-sensitive metrics like revenue counters to anyone who can reach the endpoint.
Extension challenge: Add a Histogram tracking the response payload size in bytes (cricketpulse_http_response_size_bytes) with appropriate buckets. Then write a PromQL query to find the p95 response size per endpoint. This exercises histogram bucket selection — think about the expected range of response sizes before choosing buckets.
- Centralise metric definitions in a single metrics.py module to prevent duplicate registration errors.
- Use FastAPI middleware to instrument all routes automatically — never rely on per-handler instrumentation.
- make_asgi_app() mounts the /metrics endpoint as an ASGI sub-application, compatible with any ASGI server.
- Run traffic for at least 5 minutes before querying rate() — the 5m window requires 5 minutes of data.
- Verify metrics via the Prometheus HTTP API before building Grafana dashboards on top of them.
- In production, restrict /metrics access to Prometheus IPs via network policy or basic authentication.