Tracing Practice — Trace a Request End-to-End

What You'll Build

In this exercise you will instrument a Python FastAPI service with OpenTelemetry, deploy an OTel Collector and Grafana Tempo using Docker Compose, configure Loki derived fields for log-to-trace navigation, and verify end-to-end correlation — from a Grafana metrics panel through logs to a full trace waterfall in Tempo.

Analogy🏏Cricket

🏏 Think of it like cricket: This exercise is like being handed a cricket ground that has no scoreboards, no Hawk-Eye cameras, and no stump microphones, and being told to instrument it before the match starts in 45 minutes. You will install the run-rate display (Prometheus metrics), set up the ball-by-ball commentary feed (structured logs), and mount the ball-tracking sensors (traces) — all in time for Rohit Sharma to face the first delivery. Just as a ground with no instrumentation cannot produce DRS reviews or post-match analytics, a service with no observability cannot produce root cause analysis. The insight is that this exercise simulates exactly the pressure of instrumenting a service before go-live, when adding telemetry is fastest and cheapest.

Prerequisites

Lessons 17, 18, and 19 completed.
Docker and Docker Compose installed.
The FastAPI app from Module 2 (Metrics Practice) available, or a new minimal FastAPI app.
Grafana with Prometheus and Loki already configured (from Modules 3 and 4).
Python packages: opentelemetry-sdk, opentelemetry-exporter-otlp-proto-grpc, opentelemetry-instrumentation-fastapi, opentelemetry-instrumentation-requests, opentelemetry-instrumentation-logging.

Step 1 — Deploy OTel Collector and Tempo

Add the OTel Collector and Grafana Tempo to your existing docker-compose.yaml. Tempo stores traces in local filesystem storage for this exercise. The Collector receives OTLP from your FastAPI app and forwards to Tempo.

Analogy🏏Cricket

🏏 Think of it like cricket: Before the first ball is bowled, you set up the tracking cameras (Collector) and connect them to the match archive (Tempo). The players (your app) don't need to know where the archive is — they just perform, and the cameras handle the rest.

yaml

# Add to docker-compose.yaml
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.100.0
    command: ["--config=/etc/otelcol/config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol/config.yaml
    ports:
      - "4317:4317"   # OTLP gRPC (your app sends here)
    depends_on:
      - tempo

  tempo:
    image: grafana/tempo:latest
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
      - tempo-data:/var/tempo
    ports:
      - "3200:3200"   # Tempo HTTP API (Grafana queries here)
      - "4318:4318"   # OTLP HTTP (alternative to gRPC)

volumes:
  tempo-data:

yaml

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s
  memory_limiter:
    limit_mib: 128
    check_interval: 1s

exporters:
  otlp/tempo:
    endpoint: "tempo:4317"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/tempo]

# tempo.yaml
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces

compactor:
  compaction:
    block_retention: 24h

Step 2 — Instrument the FastAPI App

Install the OTel Python packages and add SDK initialisation to your FastAPI app. Use auto-instrumentation for HTTP and logging, and add a manual span for a key business operation to demonstrate custom span enrichment.

Analogy🏏Cricket

🏏 Think of it like cricket: The middleware approach mirrors the third umpire's role in DRS — a system-level observer that automatically reviews every decision without the on-field umpires needing to actively invoke it for each ball. Every HTTP request passes through the middleware just as every DRS-eligible decision passes through the third umpire review system. The insight is that middleware-level instrumentation is preferable to per-handler instrumentation because it cannot be accidentally omitted from a new route.

python

# Install packages
pip install opentelemetry-sdk   opentelemetry-exporter-otlp-proto-grpc   opentelemetry-instrumentation-fastapi   opentelemetry-instrumentation-requests   opentelemetry-instrumentation-logging   json-log-formatter

# main.py — SDK initialisation at the top, before app creation
import logging
import json_log_formatter
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.logging import LoggingInstrumentor

# 1. Configure JSON logging so Loki can parse trace_id
formatter = json_log_formatter.JSONFormatter()
handler = logging.StreamHandler()
handler.setFormatter(formatter)
logging.basicConfig(level=logging.INFO, handlers=[handler])
logger = logging.getLogger(__name__)

# 2. Configure tracer provider
resource = Resource.create({"service.name": "cricketpulse-api"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint="http://otel-collector:4317", insecure=True)
    )
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

# 3. Auto-instrument logging (injects trace_id into log records)
LoggingInstrumentor().instrument(set_logging_format=True)

# 4. Create app and auto-instrument FastAPI
from fastapi import FastAPI
app = FastAPI()
FastAPIInstrumentor.instrument_app(app)

@app.get("/match/{match_id}/score")
def get_score(match_id: str):
    logger.info("score request", extra={"match_id": match_id})
    with tracer.start_as_current_span("fetch_score_from_db") as span:
        span.set_attribute("match.id", match_id)
        span.set_attribute("db.system", "postgresql")
        # simulate DB call
        import time; time.sleep(0.05)
        score = {"match_id": match_id, "runs": 247, "wickets": 4}
        span.set_attribute("score.runs", score["runs"])
    return score

Step 3 — Add Tempo as a Grafana Data Source

In Grafana, add Tempo as a data source pointing to http://tempo:3200. Then configure the Loki data source derived fields so that trace IDs in log lines become clickable links to Tempo.

Analogy🏏Cricket

🏏 Think of it like cricket: Connecting Tempo to Grafana is like linking the archive system to the broadcast app. Adding derived fields is like programming the app so that every mention of a ball number in the commentary automatically hyperlinks to the video clip of that ball.

yaml

# grafana/provisioning/datasources/tempo.yaml
apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    uid: tempo-uid
    url: http://tempo:3200
    access: proxy
    jsonData:
      httpMethod: GET
      serviceMap:
        datasourceUid: prometheus-uid   # enables service map view

  # Update Loki datasource to add derived fields
  - name: Loki
    type: loki
    uid: loki-uid
    url: http://loki:3100
    access: proxy
    jsonData:
      derivedFields:
        - matcherRegex: '"trace_id":"([a-f0-9]+)"'
          name: TraceID
          url: "$${__value.raw}"
          datasourceUid: tempo-uid
          urlDisplayLabel: "View Trace in Tempo"

Step 4 — Verify End-to-End Correlation

Generate traffic to your FastAPI app, then walk the full correlation chain: from Prometheus metrics through Loki logs to a Tempo trace. Each step should flow naturally using Grafana's built-in navigation — no manual copy-pasting of IDs.

Analogy🏏Cricket

🏏 Think of it like cricket: Run the training match, then sit in the analyst suite and verify that clicking any ball in the scorecard triggers the video replay. If every link works, the broadcast system is ready for match day.

bash

# Generate traffic
for i in $(seq 1 50); do
  curl -s "http://localhost:8000/match/IPL-2025-RCB-vs-CSK/score" > /dev/null
  sleep 0.2
done

# Verify spans reaching Tempo
curl -s "http://localhost:3200/api/search?service.name=cricketpulse-api&limit=5" | jq .

# Verify trace_id in Loki logs (check container output)
docker compose logs cricketpulse-api 2>&1 | grep trace_id | head -5

# In Grafana Explore:
# 1. Query Loki: {service="cricketpulse-api"} | json | trace_id != ""
# 2. Click a log line — "View Trace in Tempo" button should appear
# 3. Click it — Tempo waterfall should open showing root span + child spans

Verify Your Work

Run the checklist below to confirm every layer of the tracing stack is functioning. All five items should pass before you mark the exercise complete.

Analogy🏏Cricket

🏏 Think of it like cricket: The verification checklist is your pre-match sensor sign-off — before the umpires certify DRS is match-ready, every component gets a green tick: cameras aligned, ball-tracking calibrated, audio synced, the review console linked to the on-field feed, and the whole chain tested on a practice delivery. Here you run the same discipline down the tracing stack: spans are being generated, context propagates across service boundaries, the exporter ships them, the backend assembles the trace, and the full request path is visible end-to-end. Just as a review system passes only when every link works — a perfect camera is useless if the audio is unsynced — your trace is trustworthy only when all five items pass, because one broken hop leaves a gap exactly where you most need causality. Just as officials refuse to rely on a half-checked review rig on match day, you do not mark the exercise complete until every layer is confirmed. The payoff: a deliberate end-to-end checklist proves the whole tracing chain works together, so on a real incident you are reading the trace, not debugging the tracer.

bash

# Verification checklist

# 1. Spans arriving in Tempo
curl -s "http://localhost:3200/api/search?service.name=cricketpulse-api"   | jq '.traces | length'
# Expect: > 0

# 2. Trace contains expected spans
TRACE_ID=$(curl -s "http://localhost:3200/api/search?service.name=cricketpulse-api&limit=1"   | jq -r '.traces[0].traceID')
curl -s "http://localhost:3200/api/traces/$TRACE_ID"   | jq '[.batches[].scopeSpans[].spans[].name]'
# Expect: ["GET /match/{match_id}/score", "fetch_score_from_db"]

# 3. Logs contain trace_id
docker compose logs cricketpulse-api 2>&1   | python3 -c "import sys,json; [print(json.loads(l).get('trace_id','MISSING')) for l in sys.stdin if '{' in l]"   | grep -v MISSING | head -3
# Expect: 32-character hex trace IDs

# 4. Tempo data source in Grafana responds
curl -s "http://admin:admin@localhost:3000/api/datasources/name/Tempo"   | jq .url
# Expect: "http://tempo:3200"

# 5. Derived fields configured on Loki
curl -s "http://admin:admin@localhost:3000/api/datasources/name/Loki"   | jq '.jsonData.derivedFields[0].name'
# Expect: "TraceID"

If spans appear in Collector logs but not in Tempo, check that the Tempo receiver is configured for gRPC on port 4317 and that the Collector exporter endpoint matches. Use 'docker compose logs tempo' and 'docker compose logs otel-collector' side-by-side to trace the handoff.

Grafana Tempo's 'Service Graph' view (enabled when linked to a Prometheus datasource) auto-generates a topology map of service dependencies from trace data. After generating 50+ traces, navigate to Explore > Tempo > Service Graph to see CricketPulse's service map.

OTel SDK with BatchSpanProcessor and OTLPSpanExporter sends traces to the Collector over gRPC port 4317.
LoggingInstrumentor automatically injects trace_id into JSON log records for Loki correlation.
Loki derived fields with a regex on trace_id create one-click 'View Trace in Tempo' links.
The full correlation chain: Prometheus metric panel → Loki log line → Tempo trace waterfall.
Use 'docker compose logs' for both Collector and Tempo simultaneously to debug span delivery issues.