How do you monitor Kubernetes with Prometheus?
Monitor Kubernetes with Prometheus using service discovery, cAdvisor, kube-state-metrics and node_exporter for full-stack cluster visibility and alerting.
Expected Interview Answer
You monitor Kubernetes with Prometheus by using its Kubernetes service discovery to auto-find targets, scraping metrics from node-level exporters (node_exporter), the container runtime (cAdvisor via the kubelet), cluster state (kube-state-metrics), and application /metrics endpoints, then querying and alerting on that data.
Prometheus's kubernetes_sd_configs dynamically discover nodes, pods, endpoints, and services so targets appear and disappear as the cluster scales. cAdvisor exposes container CPU, memory and network usage through the kubelet; kube-state-metrics exposes object state such as deployments, replicas and pod phases; node_exporter provides host metrics. Relabeling filters and annotates targets. In practice the Prometheus Operator with ServiceMonitor and PodMonitor CRDs manages this declaratively, and Alertmanager handles routing.
- Automatic target discovery as pods scale up and down
- Full-stack visibility: node, container, cluster-state and app metrics
- Declarative, GitOps-friendly config via Operator CRDs
- Powerful PromQL for cluster-wide queries and dashboards
- Native alerting on pod restarts, saturation and outages
AI Mentor Explanation
Monitoring Kubernetes with Prometheus is like a broadcast setup covering a whole stadium: cameras auto-track every player as they enter and leave the field (service discovery), pitch-side sensors report each bowler's speed (cAdvisor), the scoreboard tracks match state (kube-state-metrics), and the ground crew reports conditions (node_exporter) — together giving a complete, live picture of the game.
Step-by-Step Explanation
Step 1
Deploy Prometheus in-cluster
Run Prometheus (often via the kube-prometheus-stack Helm chart) with RBAC to read the Kubernetes API.
Step 2
Configure service discovery
Use kubernetes_sd_configs with roles node, endpoints, pod and service so targets are found automatically.
Step 3
Scrape cAdvisor and kubelet
Collect per-container CPU, memory and network metrics exposed by the kubelet's /metrics/cadvisor endpoint.
Step 4
Add kube-state-metrics
Deploy it to expose object-level state like deployment replicas, pod phases and node conditions.
Step 5
Add node_exporter
Run it as a DaemonSet to expose host-level CPU, disk, memory and filesystem metrics per node.
Step 6
Relabel and alert
Use relabel_configs to filter and annotate targets, then define PromQL alert rules routed through Alertmanager.
What Interviewer Expects
- Naming the four data sources: node_exporter, cAdvisor, kube-state-metrics, app metrics
- Understanding kubernetes_sd_configs and its roles
- The role of relabeling in filtering targets
- Awareness of the Prometheus Operator and ServiceMonitor CRDs
- How alerting is wired through Alertmanager
Common Mistakes
- Confusing cAdvisor (container metrics) with kube-state-metrics (object state)
- Manually listing static targets instead of using service discovery
- Forgetting node_exporter, missing host-level metrics
- Not configuring RBAC, so Prometheus cannot read the API
- Ignoring relabeling, leading to noisy or wrong targets
Best Answer (HR Friendly)
“You install Prometheus inside the Kubernetes cluster and let it automatically discover everything running there. It then collects metrics from the servers, the containers, and the cluster itself, so teams can see the health of the whole system and get alerted when something breaks.”
Code Example
scrape_configs:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
action: replace
target_label: __metrics_path__
regex: (.+)
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: namespaceFollow-up Questions
- What is the difference between cAdvisor and kube-state-metrics?
- How does relabel_configs work in Kubernetes service discovery?
- What roles can kubernetes_sd_configs use and when do you pick each?
- How does the Prometheus Operator's ServiceMonitor replace raw scrape configs?
- How would you monitor a custom application's /metrics endpoint in a pod?
MCQ Practice
1. Which component exposes per-container CPU and memory usage in Kubernetes?
cAdvisor, embedded in the kubelet, exposes per-container resource usage metrics.
2. What does kube-state-metrics primarily expose?
kube-state-metrics translates Kubernetes API object state into metrics, such as replica counts and pod phases.
3. How does Prometheus find targets in a dynamic cluster?
kubernetes_sd_configs query the Kubernetes API to discover targets automatically as they change.
Flash Cards
What provides container metrics in Kubernetes? — cAdvisor, embedded in the kubelet, exposing per-container CPU, memory and network usage.
What does kube-state-metrics do? — Exposes Kubernetes object state (deployments, replicas, pod phases) as Prometheus metrics.
How are targets discovered? — Via kubernetes_sd_configs with roles like node, pod, endpoints and service.
Where do host metrics come from? — node_exporter, typically run as a DaemonSet on every node.
What is relabeling used for? — Filtering, keeping/dropping, and annotating discovered targets before scraping.