How does Prometheus federation work for scaling?
Learn how Prometheus federation scales monitoring by letting a global server scrape aggregated metrics from local servers via the /federate endpoint.
Expected Interview Answer
Prometheus federation lets one Prometheus server scrape selected, usually aggregated, time series from other Prometheus servers through the special /federate endpoint, so you can scale monitoring across many servers without a single instance collecting everything.
In hierarchical federation, per-datacenter or per-team Prometheus servers each scrape their local targets, and a global Prometheus federates a small set of aggregated metrics from them for cross-cluster views. In cross-service federation, one Prometheus pulls specific metrics from a sibling. Federation is configured with a normal scrape_config pointing at /federate plus match[] selectors, and honor_labels is usually kept so the source instance labels are preserved.
- Scales monitoring beyond one server's capacity
- Gives a global view over many clusters or datacenters
- Keeps high-cardinality raw data local to each Prometheus
- Lets you federate only aggregated, low-cardinality metrics
- Isolates failures so one datacenter outage doesn't lose all data
AI Mentor Explanation
Think of a national cricket board. Each state association keeps detailed ball-by-ball records for its own matches, but it only sends up small summary sheets — total runs, wickets, run rate — to the national head office. The head office never watches every delivery itself; it federates the state summaries into one country-wide dashboard. Prometheus federation works the same way: local servers keep the raw detail, the global server pulls only the aggregates.
Step-by-Step Explanation
Step 1
Deploy local Prometheus servers
Run a Prometheus per datacenter, cluster, or team that scrapes its own targets and stores raw high-cardinality data locally.
Step 2
Aggregate with recording rules
On each local server, precompute the summary series you want to expose globally using recording rules to keep federated cardinality low.
Step 3
Add a federation scrape_config
On the global Prometheus, add a job whose metrics_path is /federate and whose target is each local server.
Step 4
Select series with match[]
Use params match[] selectors to pull only the specific aggregated metrics rather than everything the source holds.
Step 5
Preserve source labels
Set honor_labels: true so instance and job labels from the source servers are kept instead of overwritten.
Step 6
Build global dashboards
Query the global server for cross-cluster views while drilling into local servers for detailed troubleshooting.
What Interviewer Expects
- Knowledge of the /federate endpoint and match[] selectors
- Difference between hierarchical and cross-service federation
- Why only aggregated, low-cardinality metrics should be federated
- Understanding of honor_labels in a federation context
- Awareness of federation limits versus remote write for very large scale
Common Mistakes
- Trying to federate all raw series, overloading the global server
- Forgetting honor_labels and clobbering source instance labels
- Confusing federation with remote write as a long-term storage solution
- Assuming federation gives real-time full-resolution data centrally
- Not using recording rules to pre-aggregate before federating
Best Answer (HR Friendly)
“Prometheus federation is a way to scale monitoring by letting one central Prometheus collect just the summary numbers from many smaller Prometheus servers, each of which watches its own systems. The detailed data stays local while the central one gives a big-picture view across everything.”
Code Example
scrape_configs:
- job_name: 'federate'
scrape_interval: 30s
honor_labels: true
metrics_path: '/federate'
params:
'match[]':
- '{job="prometheus"}'
- '{__name__=~"job:.*"}'
static_configs:
- targets:
- 'prometheus-dc1:9090'
- 'prometheus-dc2:9090'Follow-up Questions
- When would you choose remote write over federation for scaling?
- How do recording rules help keep federated cardinality low?
- What does honor_labels do and why is it important in federation?
- What are the risks of federating high-cardinality metrics?
- How would you design a global view across five datacenters?
MCQ Practice
1. Which endpoint does a global Prometheus scrape to federate metrics from another Prometheus?
Federation is served by the special /federate endpoint, configured via metrics_path with match[] selectors.
2. What kind of metrics are best suited for federation?
Federation should pull aggregated, low-cardinality series to avoid overloading the global server.
3. Why is honor_labels: true commonly set on a federation job?
honor_labels preserves the labels coming from the source Prometheus instead of overwriting them with the scrape job's labels.
Flash Cards
What endpoint powers Prometheus federation? — The /federate endpoint, scraped by a global Prometheus with match[] selectors.
Hierarchical federation — Local Prometheus servers scrape targets; a global server federates aggregated metrics for a cross-cluster view.
Why keep federated cardinality low? — Federating all raw series overloads the global server; use recording rules to pre-aggregate.
Role of honor_labels in federation — Preserves the source instance/job labels so global data still attributes to the right server.
Federation vs remote write — Federation is for aggregated cross-cluster views; remote write ships full data to long-term storage.