Prometheus Architecture and Scraping — Observability & Monitoring

Prometheus is a pull-based time-series monitoring system built specifically for dynamic cloud environments where services start, scale, and stop constantly. Unlike push-based systems where each service sends metrics to a central server, Prometheus periodically scrapes HTTP endpoints exposed by each service, collecting the current value of every registered metric at a configurable interval. This pull model has a profound operational advantage: if a service stops sending metrics, Prometheus immediately knows it is down (the scrape fails), whereas push-based systems require separate heartbeat mechanisms to detect silence. Prometheus was built at SoundCloud in 2012 and open-sourced in 2015, inspired by Google's internal Borgmon monitoring system. It became the de facto standard for Kubernetes monitoring because its service discovery mechanisms natively understand Kubernetes pod labels, making it possible to automatically begin monitoring new services the moment they are deployed without any manual configuration change.

Analogy🏏Cricket

🏏 Think of it like cricket: During India's 2023 World Cup final against Australia, the team management tracked three distinct data streams simultaneously to understand match performance. The scoreboard showed run rate, current score, and required run rate — aggregated numbers updated every over, equivalent to metrics. The commentary and match notes recorded each delivery's outcome — Rohit Sharma edged a yorker from Hazlewood at the 12th over, first ball — equivalent to logs. The ball-tracking DRS system traced the exact path of each delivery from Bumrah's hand through the air to the stumps, showing the full journey of that dismissal — equivalent to traces. Just as the scoreboard alone cannot explain why the run rate collapsed (you need the logs to see specific wicket events and traces to follow the pressure chain from bowler to batter to fielder), metrics alone cannot explain why your API latency spiked — you need logs for individual request events and traces to follow the request across services. The insight is that each pillar answers a different question: metrics give magnitude, logs give events, and traces give causality — and you need all three to diagnose a complex failure.