Centralized Logging with Loki — Observability & Monitoring

Loki is a horizontally scalable, highly available log aggregation system designed by Grafana Labs to be cost-efficient by indexing only metadata labels rather than full log content. Unlike Elasticsearch, which indexes every word in every log line creating a large, expensive index, Loki stores log content as compressed chunks in object storage and indexes only the label set attached to each log stream. Querying Loki searches the compressed chunks using parallel scanning rather than a prebuilt index. This design makes Loki dramatically cheaper to operate at scale, at the cost of query performance on unfiltered high-cardinality queries. The correct usage pattern — always narrow by label before applying content filters — matches naturally with well-structured logging.

Analogy🏏Cricket

🏏 Think of it like cricket: During India's 2023 World Cup final against Australia, the team management tracked three distinct data streams simultaneously to understand match performance. The scoreboard showed run rate, current score, and required run rate — aggregated numbers updated every over, equivalent to metrics. The commentary and match notes recorded each delivery's outcome — Rohit Sharma edged a yorker from Hazlewood at the 12th over, first ball — equivalent to logs. The ball-tracking DRS system traced the exact path of each delivery from Bumrah's hand through the air to the stumps, showing the full journey of that dismissal — equivalent to traces. Just as the scoreboard alone cannot explain why the run rate collapsed (you need the logs to see specific wicket events and traces to follow the pressure chain from bowler to batter to fielder), metrics alone cannot explain why your API latency spiked — you need logs for individual request events and traces to follow the request across services. The insight is that each pillar answers a different question: metrics give magnitude, logs give events, and traces give causality — and you need all three to diagnose a complex failure.