Observability in Kubernetes

Analogy categoryAnalogies

The Three Pillars in a Dynamic Environment

Kubernetes makes observability harder than a static VM fleet because pods are ephemeral — a crashing container might be rescheduled to a different node with a new pod name before you can even attach a debugger. Metrics (typically scraped by Prometheus via the /metrics endpoint using the OpenMetrics format) answer 'what is happening and how much', logs (aggregated by something like Fluent Bit or Loki since a pod's local logs vanish when it's deleted) answer 'what specifically happened', and traces (collected via OpenTelemetry, propagating a trace ID across service boundaries) answer 'where in a multi-service request did the time go'. None of the three pillars alone is sufficient — a metrics dashboard can show elevated latency without telling you which specific request failed or why, which is where correlated logs and traces come in.

🏏

Cricket analogy: It's like a cricket broadcast combining the scoreboard (metrics: runs, overs, run rate), the ball-by-ball commentary (logs: exactly what happened on each delivery), and the Hawk-Eye trajectory replay (traces: the ball's full path) — no single feed alone tells the full story of a dismissal.

Instrumenting and Correlating Signals

Effective Kubernetes observability requires consistent labeling so all three signal types can be correlated by the same dimensions — namespace, pod name, container name, and ideally a request-level trace ID injected into log lines. The kube-state-metrics exporter surfaces object-level state (pod phase, deployment replica counts, PVC status) that cAdvisor-based node metrics don't capture, while OpenTelemetry Collector can receive traces, metrics, and logs through a single pipeline and route them to different backends (Tempo for traces, Prometheus for metrics, Loki for logs) using OTLP as a common protocol. Without shared labels, an SRE ends up manually correlating a Grafana panel timestamp against a completely separate log search, which is slow and error-prone during an active incident.

🏏

Cricket analogy: It's like a stadium's stat system tagging every data feed with the same match ID and over number, so the scoreboard, ball-tracking, and commentary all reference the exact same delivery instead of analysts cross-checking three separate spreadsheets during a live review.

Prometheus Alerting in Practice

yaml

groups:
  - name: checkout-service.rules
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{job="checkout-service",status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{job="checkout-service"}[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "checkout-service 5xx rate above 5% for 10m"
          runbook: "https://runbooks.internal/checkout-service-5xx"

Set alerts on symptoms users actually feel (error rate, latency percentiles, saturation) rather than raw causes (CPU usage, pod restarts) — the latter make good dashboard panels and debugging aids but are noisy and cause alert fatigue when used as page-worthy alert conditions on their own.

Metrics, logs, and traces are complementary pillars — none alone answers 'why' during an incident.
Pod ephemerality means logs must be aggregated off-node (Loki, Fluent Bit) before pods disappear.
kube-state-metrics exposes Kubernetes object state; cAdvisor/node-exporter expose resource usage.
OpenTelemetry Collector can unify traces, metrics, and logs through one pipeline using OTLP.
Consistent labels (namespace, pod, trace ID) across all signals enable fast cross-signal correlation.
Prometheus alerting rules should target user-facing symptoms, not just raw resource metrics.
for: durations in alert rules prevent flapping alerts from brief, self-resolving blips.