How does service discovery work in Prometheus?
Learn how Prometheus service discovery auto-finds scrape targets from Kubernetes, Consul, EC2 and DNS, using __meta_ labels and relabel_configs to stay current.
Expected Interview Answer
Service discovery lets Prometheus automatically find and update its list of scrape targets from an external source of truth, instead of relying on a hand-maintained static list.
Prometheus supports many discovery mechanisms (Kubernetes, Consul, EC2, DNS, file-based, and more) configured under a job's service_discovery section. Each mechanism returns a set of targets plus metadata labels prefixed with __meta_, which you reshape with relabel_configs to decide which targets to keep, what address to scrape, and which labels to attach. Prometheus refreshes discovery on an interval, so targets appearing or disappearing are picked up without a restart.
- Targets update automatically as infrastructure scales up or down
- No manual editing of target lists in dynamic environments
- Rich metadata labels enable flexible filtering with relabeling
- Works across cloud providers, orchestrators, and DNS
- Reduces configuration drift and human error
AI Mentor Explanation
Service discovery is like a team manager who never keeps a fixed printed squad list. Instead, he phones the selection board before every match and gets the current eleven, including late replacements for injured players. Prometheus does the same: rather than a frozen roster of servers, it asks a live source who is playing right now, so a newly substituted fielder is scraped for stats and a benched one is quietly dropped without redrafting the sheet.
Step-by-Step Explanation
Step 1
Choose a discovery mechanism
Pick the SD source that matches your environment, such as kubernetes_sd_configs, consul_sd_configs, ec2_sd_configs, dns_sd_configs, or file_sd_configs.
Step 2
Configure it in the job
Add the discovery block under a scrape_config job so Prometheus knows where to query for candidate targets.
Step 3
Receive targets and meta labels
Each discovered target arrives with __meta_ prefixed labels describing its source metadata, plus the __address__ label for the scrape endpoint.
Step 4
Filter with relabel_configs
Use relabeling to keep or drop targets, rewrite __address__, and promote useful meta labels into stable target labels.
Step 5
Refresh continuously
Prometheus re-queries the SD source on an interval, so appearing and disappearing targets are reflected automatically without a restart.
What Interviewer Expects
- Understanding that SD replaces static target lists in dynamic environments
- Knowledge of common mechanisms like Kubernetes, Consul, EC2, DNS, and file SD
- The role of __meta_ labels and relabel_configs
- Awareness that discovery refreshes without a restart
- How __address__ determines the scrape endpoint
Common Mistakes
- Confusing service discovery with static_configs
- Forgetting that relabel_configs are needed to filter and shape discovered targets
- Assuming targets require a Prometheus restart to update
- Not knowing that meta labels are dropped unless promoted via relabeling
- Believing SD scrapes everything it discovers regardless of keep/drop rules
Best Answer (HR Friendly)
“Service discovery is how Prometheus automatically finds the servers it should monitor instead of keeping a manual list. It asks a live source such as Kubernetes or a cloud provider which machines are running right now, so monitoring keeps up as the infrastructure grows or shrinks.”
Code Example
scrape_configs:
- job_name: 'kubernetes-pods'
kubernetes_sd_configs:
- role: pod
relabel_configs:
# Only keep pods that opt in with an annotation
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
# Use the pod's declared metrics port
- source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
action: replace
regex: ([^:]+)(?::\d+)?;(\d+)
replacement: $1:$2
target_label: __address__
# Promote the namespace into a stable label
- source_labels: [__meta_kubernetes_namespace]
target_label: namespaceFollow-up Questions
- What is the difference between static_configs and service discovery?
- How do relabel_configs differ from metric_relabel_configs?
- What are __meta_ labels and how are they used?
- How does file-based service discovery work?
- How often does Prometheus refresh discovered targets?
MCQ Practice
1. What prefix do labels attached by service discovery mechanisms use?
Discovery mechanisms attach metadata labels prefixed with __meta_, which you can inspect and promote using relabel_configs.
2. Which configuration block is used to filter and reshape discovered targets?
relabel_configs run at discovery time to keep, drop, and rewrite targets and their labels, including __address__.
3. What happens when a new instance appears in a service discovery source?
Prometheus periodically re-queries the SD source, so new targets are discovered and scraped without a restart or manual edit.
Flash Cards
What is service discovery in Prometheus? — A mechanism that automatically finds scrape targets from a live source like Kubernetes or Consul instead of a static list.
What are __meta_ labels? — Metadata labels attached to discovered targets that you reshape with relabel_configs; dropped unless promoted.
Which block filters discovered targets? — relabel_configs, which can keep, drop, and rewrite targets and the __address__ scrape endpoint.
Do new targets require a restart? — No. Prometheus refreshes discovery on an interval and picks up added or removed targets automatically.
Name common SD mechanisms. — kubernetes_sd_configs, consul_sd_configs, ec2_sd_configs, dns_sd_configs, and file_sd_configs.