How does Prometheus's pull-based metrics model work?
See how Prometheus's pull model scrapes /metrics endpoints on an interval, tracks target health with the up metric, and uses Pushgateway for batch jobs.
Expected Interview Answer
In Prometheus's pull model, the server periodically sends HTTP GET requests to each target's /metrics endpoint, reads the current metric values in a plain-text exposition format, and stores them as time-series samples — the targets never push to Prometheus.
Prometheus discovers targets through static configuration or service discovery, then scrapes each on a configured scrape_interval. Each scrape returns a snapshot of counters, gauges, histograms, and summaries with their labels; Prometheus timestamps and appends them to its local TSDB. Because the server initiates collection, it inherently knows whether a target is up (the up metric), can control scrape frequency centrally, and avoids clients needing to know the server's address. For short-lived batch jobs that vanish before a scrape, the Pushgateway bridges the gap.
- Server-side control of scrape frequency and timeouts
- Built-in target health via the up metric
- Simple targets — just expose an endpoint, no client config of the server
- Works cleanly with service discovery in dynamic environments
- Easier to detect a down target than with push
AI Mentor Explanation
The pull model is like a coach who walks to each net session and personally checks how every batter is doing at set times, rather than waiting for players to come report. Because the coach initiates each check, they instantly know if a player is absent, and they control exactly how often each net is inspected across the whole squad.
Step-by-Step Explanation
Step 1
Register targets
Targets are listed via static_configs or discovered dynamically through service discovery.
Step 2
Expose /metrics
Each target serves current metric values in the Prometheus text exposition format over HTTP.
Step 3
Scrape on interval
The server issues an HTTP GET to each target every scrape_interval within a scrape_timeout.
Step 4
Parse and label
Returned samples are parsed and enriched with target labels like job and instance.
Step 5
Store and record health
Samples are appended to the TSDB and an up metric records whether the scrape succeeded.
What Interviewer Expects
- Server initiates collection via HTTP scrapes
- Role of scrape_interval and scrape_timeout
- How the up metric signals target health
- Role of service discovery in finding targets
- When Pushgateway is needed for short-lived jobs
Common Mistakes
- Saying targets push metrics to Prometheus
- Believing Pushgateway is for all pushing, not just batch jobs
- Forgetting the up metric as a health signal
- Confusing scrape_interval with evaluation_interval
Best Answer (HR Friendly)
“Instead of each service sending its stats to Prometheus, Prometheus goes out and asks every service for its current numbers at regular intervals. This lets Prometheus stay in control of how often it checks and instantly know if a service stops responding.”
Code Example
scrape_configs:
- job_name: 'web'
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
static_configs:
- targets: ['web-1:9100', 'web-2:9100']Follow-up Questions
- When would you use the Pushgateway instead of a normal scrape?
- What does the up metric tell you and how is it generated?
- How does service discovery work with the pull model?
- What are the trade-offs of pull versus push monitoring?
- How does scrape_interval interact with rate() calculations?
MCQ Practice
1. In the pull model, who initiates metric collection?
Prometheus initiates collection by scraping each target's HTTP endpoint on a schedule.
2. What metric indicates whether a scrape succeeded?
Prometheus automatically records an up metric (1 or 0) for each scrape target.
3. Which component helps expose metrics from short-lived batch jobs?
The Pushgateway lets ephemeral jobs push metrics that Prometheus then scrapes from it.
Flash Cards
Who initiates a scrape? — The Prometheus server sends an HTTP GET to each target's /metrics endpoint.
What controls scrape frequency? — The scrape_interval setting, bounded by scrape_timeout, configured server-side.
How is target health known? — The automatically generated up metric is 1 for a successful scrape, 0 otherwise.
How do short-lived jobs report? — Via the Pushgateway, which holds their metrics for Prometheus to scrape.