What is the difference between push and pull metric collection?
Learn the difference between push and pull metric collection, why Prometheus uses pull, when to use the Pushgateway, and the trade-offs each model makes.
Expected Interview Answer
In pull collection the monitoring server periodically scrapes metrics by requesting them from each target over HTTP, whereas in push collection each target sends its metrics to the monitoring server (or a gateway) on its own schedule.
Prometheus is fundamentally pull-based: it maintains a list of targets, scrapes their /metrics endpoints at a configured interval, and treats a failed scrape as a signal that a target is down. Push systems like StatsD or Prometheus' own Pushgateway invert this, letting short-lived jobs emit data before they exit. Pull gives the server control over timing and easy target-liveness detection; push suits ephemeral batch jobs that vanish before any scrape could reach them.
- Pull gives centralized control over scrape timing and cardinality
- Pull makes target liveness trivial to detect via the up metric
- Push handles short-lived and batch jobs that die before a scrape
- Push works when targets sit behind NAT or firewalls
- Pull avoids clients overwhelming the server during incidents
AI Mentor Explanation
A pull system is like the scorer walking to each fielder at fixed overs to ask for their catch and run tallies on his own schedule, staying in full control. A push system is like each fielder shouting their own numbers to the scorer whenever they feel like it. Pull keeps the scorer's book consistent; push risks a nervous fielder flooding him with updates mid-over.
Step-by-Step Explanation
Step 1
Define the targets
In pull, list the endpoints to scrape in the server config; in push, configure each client with the server or gateway address.
Step 2
Choose the interval
Pull sets one scrape_interval centrally; push leaves timing to each client, so cadence varies per source.
Step 3
Expose or emit
Pull targets expose a /metrics endpoint; push clients serialize and send metrics over the network themselves.
Step 4
Handle ephemeral jobs
Route short-lived batch jobs through a Pushgateway so their metrics survive until Prometheus scrapes the gateway.
Step 5
Detect liveness
Pull records an up metric per scrape; push must add explicit heartbeats since a silent client is ambiguous.
What Interviewer Expects
- Clear statement that Prometheus is pull-based by default
- Understanding of the up metric for target health
- When push (Pushgateway) is the correct choice
- Awareness of firewall/NAT and cardinality trade-offs
- Not conflating Pushgateway with a general metrics buffer
Common Mistakes
- Claiming Prometheus is push-based
- Using Pushgateway for long-running services instead of scraping them directly
- Forgetting that push loses easy liveness detection
- Ignoring that push clients can overwhelm the server during incidents
Best Answer (HR Friendly)
“Pull means the monitoring server goes and fetches numbers from each service on a schedule it controls, which is how Prometheus normally works. Push means each service sends its own numbers in, which is only needed for short jobs that finish too quickly to be fetched.”
Code Example
scrape_configs:
- job_name: 'api'
scrape_interval: 15s
static_configs:
- targets: ['api-1:9100', 'api-2:9100']
# Push: scrape the Pushgateway that batch jobs write to
- job_name: 'pushgateway'
honor_labels: true
static_configs:
- targets: ['pushgateway:9091']Follow-up Questions
- Why is the Pushgateway discouraged for long-running services?
- How does Prometheus detect that a target is down?
- What does the honor_labels setting do on a Pushgateway scrape?
- How would you monitor a service behind a firewall with a pull model?
- What are the cardinality risks of a push-based system under load?
MCQ Practice
1. By default, how does Prometheus collect metrics from its targets?
Prometheus is pull-based: it scrapes each target's HTTP /metrics endpoint on a configured interval.
2. Which scenario is the intended use of the Prometheus Pushgateway?
The Pushgateway exists so ephemeral batch jobs can expose their metrics until Prometheus scrapes the gateway.
3. What advantage does pull collection give for health monitoring?
Each scrape emits an up metric, so a failed scrape immediately signals that a target is unreachable.
Flash Cards
Is Prometheus push or pull? — Pull - it scrapes targets' HTTP /metrics endpoints on a schedule it controls.
What is the Pushgateway for? — Letting short-lived batch jobs expose metrics until Prometheus scrapes the gateway.
How does pull detect a dead target? — A failed scrape sets the up metric to 0 for that target.
When does push beat pull? — Ephemeral jobs and targets behind NAT/firewalls that cannot be scraped directly.