What You'll Build
In this exercise you will build a complete Service Health Dashboard for the CricketPulse scoring service using the four-row hierarchy, add a parameterised $endpoint variable, push a deployment annotation, and share the dashboard via snapshot. By the end you will have a functional production-grade dashboard that could be used during a real incident, with all the structural patterns from Lessons 9, 10, and 11 applied in a single working artefact.
Analogy🏏Cricket
🏏 Think of it like cricket: This exercise is like being handed a cricket ground that has no scoreboards, no Hawk-Eye cameras, and no stump microphones, and being told to instrument it before the match starts in 45 minutes. You will install the run-rate display (Prometheus metrics), set up the ball-by-ball commentary feed (structured logs), and mount the ball-tracking sensors (traces) — all in time for Rohit Sharma to face the first delivery. Just as a ground with no instrumentation cannot produce DRS reviews or post-match analytics, a service with no observability cannot produce root cause analysis. The insight is that this exercise simulates exactly the pressure of instrumenting a service before go-live, when adding telemetry is fastest and cheapest.
Prerequisites
- Lessons 8 stack running (cricketpulse service and Prometheus) — docker compose ps should show both Up
- Lessons 9-11 completed — understand data sources, panel types, variables, and annotations
- Grafana basics — you should be able to create a panel and configure a query
- Pushgateway available (add to docker-compose.yml in Step 4)
Setup — Add Grafana to the Stack
Add Grafana to the existing docker-compose.yml and create the provisioning directory structure. Anonymous auth is enabled for this exercise to avoid login friction — in production, configure LDAP, OAuth, or Grafana's built-in user management.
Analogy🏏Cricket
🏏 Think of it like cricket: Adding Grafana to your docker-compose stack and creating the provisioning directories is like wheeling the giant screen into the ground and pre-loading the graphics config before the crowd arrives, so the moment the match starts the visuals are already wired to the live feeds. Enabling anonymous auth for this exercise is like leaving the practice-ground screen unlocked during a warm-up so nobody wastes time fumbling for keys — convenient for a net session, but for the real tournament you would put it behind proper access control (LDAP, OAuth, or Grafana's own user management), exactly as a stadium locks down its broadcast controls on match day. Just as pre-mounting the screen and its config means play is never delayed setting up visuals, provisioning Grafana up front means your dashboards appear automatically on container start. The payoff: getting the display infrastructure stood up and auto-provisioned first means the rest of the exercise is about the data, not fighting the tooling.
bash
# Prerequisites: Lesson 8 stack must be running
cd cricketpulse-metrics
docker compose ps # verify cricketpulse and prometheus are running
# Add Grafana to docker-compose.yml
# (append to the services section)
cat >> docker-compose.yml << 'YAML'
grafana:
image: grafana/grafana:latest
ports:
- '3000:3000'
environment:
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning
YAML
mkdir -p grafana/provisioning/datasources grafana/provisioning/dashboards
docker compose up -d grafana
# Grafana UI: http://localhost:3000
Step 1 — Provision the Prometheus Datasource
Create the datasource provisioning YAML so Grafana automatically connects to Prometheus on startup. Provisioning ensures the datasource survives container restarts without manual reconfiguration.
Analogy🏏Cricket
🏏 Think of it like cricket: Provisioning the datasource is like the ICC officially designating Hawk-Eye as the DRS technology provider before the match begins. Without this formal designation, the umpires cannot use Hawk-Eye for review decisions, even though Hawk-Eye is running and perfectly functional. Without the Grafana datasource configuration, panels cannot query Prometheus, even though Prometheus is running and perfectly functional. The provisioning file is the formal connection between the two systems.
yaml
# grafana/provisioning/datasources/datasources.yml
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
uid: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
jsonData:
timeInterval: '15s'
# After saving, restart Grafana to pick up the provisioned datasource:
docker compose restart grafana
# Verify datasource via API:
curl http://localhost:3000/api/datasources | python3 -m json.tool
Step 2 — Build the Four-Row Dashboard
Create a new dashboard in the Grafana UI and add panels following the four-row hierarchy. Build the Status row first with three Stat panels, then the Trends row with two Time Series panels, then the Breakdown row with a Table panel. Use the PromQL expressions in the code block as a reference.
Analogy🏏Cricket
🏏 Think of it like cricket: Building the dashboard in the correct row order mirrors how a broadcast production team assembles the match overlay. They build the score ticker first (Status row — the element that will be most-viewed most often), then the run rate chart (Trends row), then the batting scorecard (Breakdown row). Starting with the most important element ensures it works perfectly before adding lower-priority panels. An error rate Stat panel that shows no data or incorrect thresholds is more damaging than a Table panel that is incomplete, because the Stat panel is the first thing an incident responder looks at.
bash
# Dashboard creation steps (Grafana UI walkthrough)
# 1. Click + -> New Dashboard -> Add visualization
# ROW 1: STATUS (Stat panels)
# Panel A: Error Rate
# Type: Stat
# Query: sum(rate(cricketpulse_http_requests_total{status_code=~'5..'}[5m]))
# / sum(rate(cricketpulse_http_requests_total[5m]))
# Unit: Percent (0.0-1.0)
# Thresholds: green=0, yellow=0.001, red=0.01
# Panel B: p99 Latency
# Type: Stat
# Query: histogram_quantile(0.99,
# sum(rate(cricketpulse_http_request_duration_seconds_bucket[5m])) by (le)
# )
# Unit: seconds (s)
# Thresholds: green=0, yellow=0.5, red=1.0
# Panel C: Request Rate
# Type: Stat
# Query: sum(rate(cricketpulse_http_requests_total[5m]))
# Unit: requests/sec (reqps)
# ROW 2: TRENDS (Time Series)
# Panel D: Request Rate Over Time
# Type: Time Series
# Query: sum(rate(cricketpulse_http_requests_total[5m])) by (endpoint)
# Legend: {{endpoint}}
# Panel E: Error Rate Over Time
# Type: Time Series
# Query: sum(rate(cricketpulse_http_requests_total{status_code=~'5..'}[5m])) by (endpoint)
# / sum(rate(cricketpulse_http_requests_total[5m])) by (endpoint)
# Unit: Percent (0.0-1.0)
# ROW 3: BREAKDOWN (Table)
# Panel F: Endpoint Request Rates
# Type: Table
# Query: sum(rate(cricketpulse_http_requests_total[5m])) by (endpoint, status_code)
# Instant: true (show current value, not time series)
Step 3 — Add the $endpoint Variable
Add a query variable that populates from the Prometheus label values and update the Trends row panels to filter by the variable selection. Test the variable by selecting individual endpoints and verifying that panels filter correctly.
Analogy🏏Cricket
🏏 Think of it like cricket: Adding the $endpoint variable is like the broadcast team adding a bowler-filter to the bowling analysis graphic. Instead of showing all bowlers' economy rates in a single crowded chart, the graphic allows the commentator to select a specific bowler and see only that bowler's data. When an incident is affecting only the /live_score endpoint, the variable allows the responder to filter all panels to show only that endpoint's data, removing the noise of the /scorecard and /health endpoints that are functioning normally.
bash
# Add $endpoint variable to the dashboard
# Dashboard Settings -> Variables -> Add variable
#
# Name: endpoint
# Type: Query
# Datasource: Prometheus
# Query: label_values(cricketpulse_http_requests_total, endpoint)
# Multi-value: true
# Include All: true (regex: .*)
# Refresh: On time range change
#
# Update Panel D query to use the variable:
# sum(rate(cricketpulse_http_requests_total{endpoint=~'$endpoint'}[5m])) by (endpoint)
#
# Update Panel E query:
# sum(rate(cricketpulse_http_requests_total{endpoint=~'$endpoint',status_code=~'5..'}[5m])) by (endpoint)
# / sum(rate(cricketpulse_http_requests_total{endpoint=~'$endpoint'}[5m])) by (endpoint)
Step 4 — Push a Deployment Annotation
Add Pushgateway to the docker-compose stack, create the deployment annotation shell script, and configure a Grafana annotation query to display deployment events as vertical lines on the Time Series panels. Push a test annotation and verify it appears on the dashboard.
Analogy🏏Cricket
🏏 Think of it like cricket: Pushing a deployment annotation is like the moment the third umpire flashes a marker onto the timeline the instant a substitution or bowling change happens, so anyone reading the graph later knows exactly when the game state shifted. You add Pushgateway to the stack — the holding point where short-lived jobs drop their event, like the fourth official relaying a change to the scorers' desk — then a shell script fires the annotation and a Grafana annotation query draws it as a vertical line across your Time Series panels. Just as a marked 'new bowler at over 12' line lets an analyst instantly connect a change in run rate to its cause, a deploy line lets you connect a latency or error shift to the exact release that triggered it. Just as pushing the marker to the desk lets a batch job announce a one-off event a scraper would otherwise miss, Pushgateway captures the ephemeral deploy that Prometheus's pull model would never catch. The payoff: practising this wires cause directly onto your graphs, so future incidents come with their timeline already annotated.
bash
# push_deployment.sh — run this after each deployment
#!/bin/bash
VERSION=${1:-v1.0.0}
PUSHGATEWAY=http://localhost:9091
# Add Pushgateway to docker-compose.yml if not already present
# pushgateway:
# image: prom/pushgateway:latest
# ports:
# - '9091:9091'
cat <<EOF | curl --silent --data-binary @- \
${PUSHGATEWAY}/metrics/job/deployments/instance/cricketpulse
# HELP deployment_timestamp Unix timestamp of deployment
# TYPE deployment_timestamp gauge
deployment_timestamp{service="cricketpulse",version="${VERSION}"} $(date +%s)
EOF
echo "Deployment annotation pushed for cricketpulse ${VERSION}"
# In Grafana: Dashboard Settings -> Annotations -> Add annotation query
# Datasource: Prometheus
# Query: changes(deployment_timestamp{service='cricketpulse'}[1m]) > 0
# Step: 60s
# Title field: version
# Use value as timestamp: false
bash
# Verify dashboard panels return data
# 1. Generate traffic
python3 generate_traffic.py
# 2. Check each panel in Grafana (inspect -> Query -> Run)
# All panels should show non-zero values after 5 min of traffic
# 3. Test the $endpoint variable
# - Select 'live_score' only -> panels filter to live_score
# - Select 'All' -> panels show all endpoints
# 4. Push a fake deployment annotation
bash push_deployment.sh v2.0.0
# Verify: a vertical annotation line appears on Time Series panels
# 5. Generate a share URL
# Dashboard -> Share -> Link -> Copy link with current time range
# Paste the URL into an incognito window — should show the same view
# 6. Create a snapshot for this exercise
# Share -> Snapshot -> Publish
# Copy the snapshot URL — this is what you would share in an incident report
Warning: The GF_AUTH_ANONYMOUS_ENABLED=true configuration is for development only. Never deploy Grafana with anonymous admin access in production. Use GF_SECURITY_ADMIN_PASSWORD for local development and configure OIDC or SAML for production authentication. An unprotected Grafana instance exposes all your metric data and allows anyone to create alert silences or modify dashboards.
Extension challenge: Export the dashboard JSON (Dashboard Settings -> JSON Model -> Copy to clipboard) and save it to grafana/dashboards/cricketpulse-health.json. Add a dashboard provisioning configuration to grafana/provisioning/dashboards/dashboards.yml so the dashboard is automatically loaded on Grafana startup. Verify by destroying and recreating the Grafana container — the dashboard should reappear automatically.
- Provisioned datasources survive container restarts — never configure production datasources only through the UI.
- Build the Status row first — it is the most-viewed element during incidents and should be verified first.
- Test the $endpoint variable by selecting individual endpoints and verifying panel filtering works correctly.
- Deployment annotations require Pushgateway — Prometheus cannot accept pushed metrics without it.
- Always verify annotations appear on the Time Series panels before using the dashboard in production.
- Export and version-control the dashboard JSON to enable GitOps and disaster recovery.