Observability — Prometheus, Grafana and distributed tracing with OpenTelemetry

Observability is the property of a system that allows engineers to understand its internal state from its external outputs — not just whether it is up or down but why a specific request failed, which downstream service caused a latency spike, and whether the degradation in the API's p99 response time that started at 14:23 correlates with the PostgreSQL connection pool exhaustion that Prometheus captured at 14:22. Three complementary signals make a system observable: metrics (aggregated numerical measurements over time, such as request rate, error rate, and latency percentiles), logs (structured text records of discrete events that carry contextual fields enabling filtering and correlation), and traces (the end-to-end journey of a single request through multiple services, showing where time was spent and where errors occurred). Without all three, diagnosing production incidents requires guesswork at the exact moment when precision matters most — during an outage with users affected and engineers under pressure.

The Kubernetes-native observability stack consists of Prometheus for metrics collection and alerting, Grafana for dashboards and alert routing, the OpenTelemetry Collector for trace and log collection, and Jaeger or Tempo for trace storage and visualisation. These components interact through standardised interfaces: Prometheus scrapes metrics from any endpoint exposing the OpenMetrics format, Grafana queries Prometheus via PromQL, and OpenTelemetry instruments both application code and infrastructure with a single SDK that exports signals to any backend without vendor lock-in. The entire stack runs as Kubernetes workloads and can be installed as a production-ready bundle via the `kube-prometheus-stack` Helm chart, which packages Prometheus, Grafana, Alertmanager, and a complete set of Kubernetes system dashboards as a single coherent observability platform.

Analogy🏏Cricket

🏏 Think of it like cricket: The Pod-ReplicaSet-Deployment hierarchy maps precisely onto the three levels of IPL franchise team management. A Pod is a single player on the field at a given moment — the smallest unit of participation, carrying its own identity and fulfilling a specific role in the current game. A ReplicaSet is the franchise's match-day playing XI contract — it specifies that exactly eleven players matching a specific profile must always be on the field; if one is injured and leaves, the team management immediately sends a substitute of the same profile to restore the count. A Deployment is the franchise's season-long team strategy — it manages how the playing XI evolves between matches: when a new batting approach is adopted, the Deployment replaces the old XI with the new one in a controlled rolling substitution rather than swapping all eleven players simultaneously and disrupting team cohesion. Just as the franchise director does not manage individual players directly — the playing XI contract (ReplicaSet) handles the count and the season strategy (Deployment) handles the transitions — you never manage Pods directly in production; the Deployment manages the transition and the ReplicaSet maintains the count. This reveals why the three-level hierarchy exists rather than one omnibus 'workload' object: each level solves one specific problem, and composing three focused abstractions produces better separation of concerns than one object that conflates scheduling, scaling, and update management.