What is Prometheus and How Does it Collect Metrics?
Learn what Prometheus is, how it pulls metrics via scraping, labels, PromQL, and Alertmanager — with a clear DevOps interview answer.
Expected Interview Answer
Prometheus is an open-source monitoring and alerting toolkit that pulls time-series metrics from configured targets over HTTP at regular intervals, stores them in its own efficient local time-series database, and lets you query and alert on them using PromQL.
Instead of applications pushing metrics to a central collector, Prometheus uses a pull model: each target exposes a `/metrics` HTTP endpoint in a plain text exposition format, and the Prometheus server scrapes that endpoint on a configured interval, tagging every sample with a timestamp and labels. Labels are key-value pairs attached to each metric series (like `instance`, `job`, or `status_code`), which turns a single metric name into a multi-dimensional dataset that PromQL can slice, aggregate, and rate over time. Because scraping is centrally controlled, Prometheus can detect a target going silent (a failed scrape) as a signal in itself, and short-lived jobs that cannot be scraped push through an intermediary called the Pushgateway. Alertmanager consumes alerting rules evaluated by the Prometheus server and handles deduplication, grouping, silencing, and routing notifications to channels like Slack or PagerDuty.
- Pull-based scraping keeps target configuration centralized and simple
- PromQL enables powerful multi-dimensional aggregation and rate calculations
- Service discovery integrates natively with Kubernetes and cloud providers
- Alertmanager decouples alert routing from the metrics storage engine
AI Mentor Explanation
Prometheus is like a team analyst who walks to every player on the boundary rope every few minutes with a clipboard, asking for their current heart rate and step count rather than waiting for players to radio it in whenever they feel like it. Each reading gets tagged with the player name and position, the exact same metric shape, so the analyst can later ask “what was the average heart rate for all fast bowlers in the last hour.” If a player does not answer when approached, that silence itself is logged as a missed check-in, flagging a possible injury. The head coach only gets paged through a separate alert desk if the readings cross a dangerous threshold, so nobody is buried in raw numbers.
Step-by-Step Explanation
Step 1
Instrument the target
The application or an exporter exposes metrics on a `/metrics` HTTP endpoint in Prometheus exposition format.
Step 2
Configure a scrape job
prometheus.yml lists targets or a service discovery mechanism, plus the scrape interval.
Step 3
Prometheus pulls and stores
The server scrapes each endpoint on schedule and stores labeled, timestamped samples in its local TSDB.
Step 4
Query and alert
PromQL powers dashboards and alerting rules; Alertmanager routes firing alerts to notification channels.
What Interviewer Expects
- Understanding that Prometheus pulls metrics rather than receiving pushed events
- Knowledge of labels as the mechanism for multi-dimensional queries
- Awareness of the Pushgateway exception for short-lived/batch jobs
- Ability to explain the separation between Prometheus (storage/eval) and Alertmanager (routing)
Common Mistakes
- Describing Prometheus as a push-based system by default
- Confusing the Pushgateway with the normal scraping model
- Not mentioning labels when explaining how PromQL aggregates data
- Treating Alertmanager and Prometheus as a single monolithic tool
Best Answer (HR Friendly)
“Prometheus is a monitoring tool that actively reaches out and collects numbers — like CPU usage or request counts — from our services on a schedule, rather than waiting for services to send that data in. It stores those numbers over time so we can query trends and set up alerts, and a companion tool called Alertmanager makes sure the right person gets notified through Slack or PagerDuty when something crosses a threshold.”
Code Example
global:
scrape_interval: 15s
scrape_configs:
- job_name: "api-service"
static_configs:
- targets: ["api-service:9100"]
metrics_path: /metrics
- job_name: "kubernetes-pods"
kubernetes_sd_configs:
- role: podsum(rate(http_requests_total{job="api-service", status_code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api-service"}[5m]))Follow-up Questions
- Why does Prometheus use a pull model instead of a push model?
- What is the Pushgateway for, and when should you avoid using it?
- How does PromQL rate() differ from irate()?
- How does Prometheus handle long-term storage and high availability?
MCQ Practice
1. How does Prometheus typically collect metrics from a target?
Prometheus uses a pull model, scraping an HTTP endpoint on each target on a configured interval.
2. What is the role of labels in a Prometheus metric?
Labels attach key-value dimensions to a metric series, letting PromQL filter and aggregate across dimensions like instance or status code.
3. When would you use the Prometheus Pushgateway?
The Pushgateway is a workaround for ephemeral jobs that would otherwise disappear before Prometheus could scrape them directly.
Flash Cards
How does Prometheus get its data? — It pulls (scrapes) metrics from a /metrics HTTP endpoint on each target at a set interval.
What are Prometheus labels for? — Key-value pairs that add dimensions to a metric, enabling flexible filtering and aggregation in PromQL.
What handles alert notification routing? — Alertmanager — it deduplicates, groups, and routes firing alerts to channels like Slack or PagerDuty.
What is the Pushgateway used for? — Letting short-lived/batch jobs push a final metric snapshot since they cannot be scraped directly.