Why Observability Is Different in Kubernetes
Pods are ephemeral: they are created, rescheduled, and destroyed constantly by the scheduler and controllers. Traditional host-based monitoring and log files on disk don't work well here, so Kubernetes observability relies on centralized log aggregation and a metrics pipeline that can track short-lived, dynamically-named objects.
Cricket analogy: Pods being ephemeral is like players constantly being rotated in and out of the squad for fitness reasons; you can't rely on tracking one fixed player's stats over the season, so the board needs a centralized statistics system tracking the whole rotating squad.
Quick inspection with kubectl
For ad-hoc debugging, kubectl logs streams a container's stdout/stderr, and kubectl top shows live CPU/memory usage sourced from the metrics-server; kubectl top requires the metrics-server add-on and only reports current usage, not history. These are the first tools to reach for before setting up a full monitoring stack.
Cricket analogy: kubectl logs streaming live commentary is like tuning into ball-by-ball radio coverage, while kubectl top showing live CPU/memory is like a fitness tracker showing a player's current heart rate — useful in the moment, but you need the metrics-server 'sensor' installed and it shows no history.
kubectl logs pod/web-7d9f8c-abcde # current container logs
kubectl logs pod/web-7d9f8c-abcde -c sidecar # specific container in a multi-container pod
kubectl logs pod/web-7d9f8c-abcde --previous # logs from the last crashed instance
kubectl logs -f deployment/web --tail=100 # follow, tail last 100 lines
kubectl top nodes
kubectl top pods -n prod --sort-by=memoryMetrics with Prometheus/Grafana, and centralized logging
Prometheus is the standard metrics system for Kubernetes: it scrapes /metrics HTTP endpoints on a pull basis, stores time-series data, and evaluates alerting rules. kube-state-metrics exposes cluster object state (Deployment replica counts, Pod phases, etc.) in Prometheus format, complementing node/cAdvisor metrics about resource usage, and Grafana then visualizes this data in dashboards. A scrape target must be discoverable via a Service/Endpoints or ServiceMonitor and expose metrics on a stable port; short-lived batch Jobs often need a Pushgateway instead. Because Pods are ephemeral, logs must also be shipped off-node: a common pattern runs a log-forwarder (Fluent Bit or Fluentd) as a DaemonSet, reading each node's container log files and shipping them to a backend such as Elasticsearch/OpenSearch or Loki, queried via Kibana or Grafana.
Cricket analogy: Prometheus scraping /metrics endpoints is like a statistician regularly walking the boundary collecting live stats from every fielder; kube-state-metrics is the official scoreboard showing squad composition, Grafana is the big screen visualizing it, and short overs (batch Jobs) push their totals via a runner (Pushgateway) instead, while a log-forwarder DaemonSet is like a stadium announcer relaying every ground's commentary to a central broadcast archive.
# Example ServiceMonitor (Prometheus Operator) scraping an app's /metrics endpoint
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: web-metrics
labels:
release: prometheus
spec:
selector:
matchLabels:
app: web
endpoints:
- port: metrics
path: /metrics
interval: 30sEvents as an observability signal, and alerting
kubectl describe pod shows recent Events (scheduling failures, probe failures, image pull errors) which are often the fastest way to diagnose a broken Pod, complementing logs and metrics. Prometheus Alertmanager evaluates alerting rules (e.g. high error rate, Pod CrashLoopBackOff for N minutes, node disk pressure) and routes notifications to Slack, PagerDuty, or email, with grouping and silencing to reduce noise.
Cricket analogy: kubectl describe pod showing recent Events is like the third umpire's replay log showing exactly why a run-out decision was made — the fastest way to diagnose a controversial call; Alertmanager is like the ground staff paging the medical team when a player shows signs of injury, routing the alert with grouping to avoid spamming everyone.
kubectl describe pod web-7d9f8c-abcde
# ...
# Events:
# Type Reason Age From Message
# Warning Unhealthy 2m kubelet Readiness probe failed: HTTP probe failed with statuscode: 503
# Warning BackOff 1m kubelet Back-off restarting failed container- kubectl logs shows a single container's stdout/stderr; --previous recovers logs from a crashed instance.
- kubectl top needs metrics-server and only shows point-in-time usage, not history.
- Prometheus scrapes metrics on a pull model; kube-state-metrics adds cluster object state.
- Grafana visualizes Prometheus (and other) data sources as dashboards.
- Log shippers (Fluent Bit/Fluentd) typically run as a DaemonSet to centralize logs off ephemeral Pods.
- kubectl describe pod's Events section is often the fastest way to see why a Pod is unhealthy.
- Alertmanager turns Prometheus alert rules into routed, deduplicated notifications.
Practice what you learned
1. What add-on does kubectl top rely on to report Pod and Node resource usage?
2. How does Prometheus typically collect metrics from applications?
3. Why is a log-forwarding DaemonSet a common pattern in Kubernetes logging?
4. Which kubectl command shows recent scheduling and probe-failure events for a Pod?
5. What is kube-state-metrics used for?
Was this page helpful?
You May Also Like
Health Checks and Restart Policies
How Docker uses HEALTHCHECK instructions and restart policies to detect unhealthy containers and automatically recover from failures.
Horizontal Pod Autoscaling
Learn how the HorizontalPodAutoscaler automatically adjusts replica counts based on CPU, memory, or custom metrics to match workload demand.
kubectl Basics
A practical introduction to kubectl, the command-line tool used to interact with a Kubernetes cluster's API server.
Kubernetes Security Basics
Learn defensive Kubernetes security fundamentals: RBAC, NetworkPolicies, Pod security context, secrets handling, and image scanning.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsApache Kafka Study Notes
Kafka · 30 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics