Logging Practice — Centralize App Logs — Observability & Monitoring

What You'll Build

In this exercise you will add JSON-structured logging to the CricketPulse FastAPI service, deploy Loki and Promtail via docker-compose with Docker socket log collection, add the Loki datasource to Grafana, run LogQL queries against the collected logs, and verify the split-mode metric-to-log correlation workflow in Grafana Explore. By the end you will have a working three-stack observability setup: metrics in Prometheus, logs in Loki, both visible in Grafana.

Analogy🏏Cricket

🏏 Think of it like cricket: This exercise is like being handed a cricket ground that has no scoreboards, no Hawk-Eye cameras, and no stump microphones, and being told to instrument it before the match starts in 45 minutes. You will install the run-rate display (Prometheus metrics), set up the ball-by-ball commentary feed (structured logs), and mount the ball-tracking sensors (traces) — all in time for Rohit Sharma to face the first delivery. Just as a ground with no instrumentation cannot produce DRS reviews or post-match analytics, a service with no observability cannot produce root cause analysis. The insight is that this exercise simulates exactly the pressure of instrumenting a service before go-live, when adding telemetry is fastest and cheapest.

Prerequisites

Lesson 12 stack running (CricketPulse + Prometheus + Grafana) — docker compose ps showing all three Up
Lessons 13-15 completed — understand Loki architecture, structured logging, and LogQL
Docker socket accessible — Promtail needs /var/run/docker.sock for container discovery
python-json-logger installed in the virtual environment

Setup — Add Loki and Promtail to the Stack

Extend the docker-compose.yml with Loki and Promtail services and create their configuration directories. The Docker socket volume mount gives Promtail access to container metadata and log streams without requiring any changes to the application containers.

Analogy🏏Cricket

🏏 Think of it like cricket: Extending docker-compose with Loki and Promtail is like installing a central commentary archive plus a set of feed-collectors around the ground, without asking a single camera operator to change what they do. The Docker socket volume mount gives Promtail read access to every container's metadata and log stream — like giving the archive room a direct tap into all the existing camera feeds through the ground's wiring, rather than sending a runner to each position. Just as tapping the existing feeds means no camera crew has to re-cable or alter their setup, mounting the socket means no application container needs any code change to have its logs collected. Just as one central collection point beats twelve operators handing over tapes by hand, Promtail scraping the socket beats bolting a log-shipper into every service. The payoff: standing up collection infrastructure that reads from the shared socket gives you fleet-wide log capture with zero changes to the apps being observed.

bash

# Starting from the Lesson 12 stack (Grafana + Prometheus + CricketPulse)
cd cricketpulse-metrics
pip install python-json-logger --break-system-packages

# Add Loki and Promtail to docker-compose.yml
# (append to services section)
cat >> docker-compose.yml << 'YAML'
  loki:
    image: grafana/loki:latest
    ports:
      - '3100:3100'
    command: -config.file=/etc/loki/local-config.yaml
    volumes:
      - ./loki/loki-config.yml:/etc/loki/local-config.yaml
      - loki-data:/loki

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail/promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml

volumes:
  loki-data:
YAML

mkdir -p loki promtail

Step 1 — Add JSON Structured Logging

Create the logging_config.py module and update main.py to emit structured JSON logs with the mandatory fields (service, levelname, request_id, path, status_code, duration_ms). Test locally that log lines are valid JSON before deploying — the Promtail JSON pipeline stage will silently skip lines that are not valid JSON.

Analogy🏏Cricket

🏏 Think of it like cricket: Adding structured logging to the FastAPI service is like the match scorer switching from handwritten notes in a personal shorthand to the ICC's electronic ball-by-ball input form. Both record the same events, but the electronic form produces structured records that can be automatically processed by the ICC database, while the handwritten notes require manual transcription. The CricketPulseFormatter ensures every log line contains the required ICC fields (service, levelname) automatically, just as the electronic form enforces the mandatory fields before allowing submission.

python

# app/logging_config.py
import logging, json, sys
from pythonjsonlogger import jsonlogger

class CricketPulseFormatter(jsonlogger.JsonFormatter):
    def add_fields(self, log_record, record, message_dict):
        super().add_fields(log_record, record, message_dict)
        log_record['service'] = 'cricketpulse'
        log_record['env'] = 'dev'
        log_record['levelname'] = record.levelname

def configure_logging():
    handler = logging.StreamHandler(sys.stdout)
    handler.setFormatter(CricketPulseFormatter(
        fmt='%(asctime)s %(name)s %(levelname)s %(message)s'
    ))
    root = logging.getLogger()
    root.handlers = [handler]
    root.setLevel(logging.INFO)

# app/main.py — update imports and add logging middleware
from app.logging_config import configure_logging
import logging, time, uuid, random

configure_logging()
logger = logging.getLogger('cricketpulse.http')

@app.middleware('http')
async def logging_middleware(request: Request, call_next):
    request_id = str(uuid.uuid4())[:8]
    start = time.time()
    try:
        response = await call_next(request)
        logger.info('request', extra={
            'request_id': request_id,
            'method': request.method,
            'path': str(request.url.path),
            'status_code': response.status_code,
            'duration_ms': round((time.time() - start) * 1000, 2)
        })
        return response
    except Exception as exc:
        logger.error('request_error', extra={
            'request_id': request_id,
            'path': str(request.url.path),
            'error': str(exc)
        })
        raise

Step 2 — Configure Loki

Create the Loki configuration file defining the schema, storage backend, and ingestion limits. The single-binary mode configuration shown here is appropriate for development and small production deployments. The reject_old_samples_max_age: 168h (7 days) prevents accidentally ingesting historical logs that would cause Loki to allocate index entries outside the active window.

Analogy🏏Cricket

🏏 Think of it like cricket: Configuring Loki's schema, storage backend, and ingestion limits is like setting the archive's filing rules before the season — how footage is indexed, where the reels are stored, and what will and won't be accepted. Running single-binary mode suits a domestic fixture or a development ground; a global tournament would shard the roles out, just as a small ground runs one control room while a World Cup venue splits ingest, storage, and query across teams. The reject_old_samples_max_age of 168h (7 days) is the rule that the archive refuses footage older than a week — like a scoring desk that won't accept a retro-dated scorecard, because back-dating entries would force the index to allocate slots outside the active window and bloat the whole system. Just as a filing clerk who accepted arbitrarily old sheets would scatter the index and slow every lookup, ingesting historical logs outside the active window wrecks Loki's index efficiency. The payoff: getting schema, storage, and ingestion limits right up front is what keeps the log store fast and bounded as data pours in.

yaml

# loki/loki-config.yml
auth_enabled: false
server:
  http_listen_port: 3100
ingester:
  chunk_idle_period: 5m
  max_chunk_age: 1h
schema_config:
  configs:
    - from: 2020-01-01
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks
limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
chunk_store_config:
  max_look_back_period: 0s
table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

Step 3 — Configure Promtail and Deploy

Create the Promtail configuration to collect logs from all Docker containers via the Docker socket, promote service and levelname to Loki labels via relabelling and pipeline stages, and push to Loki. The Docker service discovery automatically finds all running containers and updates when containers start or stop.

Analogy🏏Cricket

🏏 Think of it like cricket: Promtail's Docker socket discovery is like the ICC's automatic ground connectivity system — when a new cricket ground is added to the ICC calendar, the ground's scoring system is automatically discovered and connected to the central database without manual configuration. Promtail does the same: when a new container starts (a new service is deployed), Promtail automatically discovers it and starts collecting its logs without requiring any configuration change. The relabelling pipeline is like the ICC's field mapping — it maps the ground's local field names to the ICC standard field names.

bash

# promtail/promtail-config.yml
server:
  http_listen_port: 9080

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_label_com_docker_compose_service']
        target_label: service
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'     # strip leading slash
        target_label: container
      - source_labels: ['__meta_docker_container_log_stream']
        target_label: stream
    pipeline_stages:
      - json:
          expressions:
            levelname: levelname
      - labels:
          levelname:

# Start the updated stack
docker compose up -d
docker compose logs promtail --tail=20
# Should see: 'Discovered new Docker target: cricketpulse'

Step 4 — Query Logs and Add to Grafana

Verify that logs are flowing into Loki using the API, then run LogQL queries in Grafana Explore to find errors and slow requests. Add the Loki datasource to Grafana provisioning and add a Logs panel to the service health dashboard. Finally, verify the split-mode correlation workflow.

Analogy🏏Cricket

🏏 Think of it like cricket: This step is the payoff of the whole session — first you confirm the archive is actually recording by checking the Loki API, like verifying the commentary feed is landing in storage before you rely on it. Then you run LogQL in Grafana Explore to find errors and slow requests, the way an analyst queries the ball-by-ball archive for exactly the deliveries that went for boundaries or wickets. You wire the Loki datasource into Grafana provisioning and add a Logs panel to the service health dashboard, mounting the commentary archive right beside the run-rate screen. Finally you verify the split-mode correlation workflow — metrics on one pane, logs on the other, synced to one window — exactly like lining up ball-tracking and audio on the same delivery for a review. Just as an archive nobody can query is worthless, logs you cannot search and correlate are just expensive noise. The payoff: proving end-to-end that logs flow in, answer LogQL queries, and correlate with metrics is what turns raw collection into a usable diagnostic tool.

bash

# 1. Generate traffic including some errors
python3 generate_traffic.py

# 2. Query Loki API directly to confirm logs are ingested
curl -G 'http://localhost:3100/loki/api/v1/query' \
  --data-urlencode 'query={service="cricketpulse"}' \
  --data-urlencode 'limit=5' | python3 -m json.tool

# 3. In Grafana Explore, query the logs:
# {service='cricketpulse'} | json | status_code >= 400
# {service='cricketpulse'} | json | duration_ms > 100

# 4. Add Loki datasource to Grafana provisioning
# (add to grafana/provisioning/datasources/datasources.yml)
cat >> grafana/provisioning/datasources/datasources.yml << 'YAML'
  - name: Loki
    type: loki
    uid: loki
    access: proxy
    url: http://loki:3100
YAML
docker compose restart grafana

# 5. Add a Logs panel to the existing dashboard:
#   Type: Logs
#   Datasource: Loki
#   Query: {service='cricketpulse', levelname='ERROR'}
#   Position: below Row 2 Trends

# 6. Verify split-mode correlation:
#   Explore -> Split -> Left=Prometheus error rate, Right=Loki error logs
#   Generate traffic -> watch both panels update

Warning: The Promtail Docker socket mount (/var/run/docker.sock:/var/run/docker.sock) gives Promtail root-equivalent access to the Docker daemon. In production, use a more restrictive log collection approach such as Docker log file tailing (/var/lib/docker/containers/*/...-json.log) or a dedicated log collector like Fluent Bit that does not require Docker socket access. Docker socket access allows the container to stop, start, or remove any container on the host.

Extension challenge: Add a LogQL alert rule to Loki's ruler component that fires when the rate of ERROR log lines exceeds 1 per second for any service. This requires adding ruler configuration to loki-config.yml and a rule file under /loki/rules/. Compare the alert firing time against the Prometheus HTTP error rate alert to verify that log-based alerts fire earlier for errors that occur before an HTTP response is returned.

Install python-json-logger and use a custom Formatter to ensure all log lines are valid JSON.
Promtail's Docker socket discovery automatically finds new containers without configuration changes.
The pipeline_stages section promotes JSON fields to Loki labels at collection time.
Verify Loki ingestion via the API before querying in Grafana — an empty result in Grafana could mean no data or a datasource misconfiguration.
Split-mode Explore links Prometheus metrics and Loki logs via shared time range selection.
Docker socket access in Promtail is a security concern — use file-based log tailing in production.