How Would You Design a Metrics Monitoring System (like Prometheus)?

System design guide to building a Prometheus-style metrics system: scraping, time-series storage, and alerting.

mediumQ59 of 224 in System Design Est. time: 5 minsLast updated: July 2026

Expected Interview Answer

Design a metrics monitoring system around periodic pull-based (or push-based via a gateway) scraping of numeric time series from every service, storage in a time-series database optimized for append-heavy writes and range queries, and a separate alerting layer that evaluates rules against that data and fires notifications on threshold breaches.

Each service exposes a metrics endpoint reporting counters, gauges, and histograms; a central scraper polls these endpoints on a fixed interval (e.g., every 15 seconds) and writes each sample as a labeled time series (metric name plus key-value dimensions like service, instance, region) into a time-series database designed for high write throughput and efficient compression of mostly-monotonic or slowly-changing values. For ephemeral or batch jobs that do not live long enough to be scraped, a push gateway lets them push a final value that the scraper then picks up. Queries against this store use a time-series query language to aggregate and downsample series (e.g., rate of a counter over 5 minutes, p99 of a histogram) for both dashboards and alerting. A separate rule evaluator runs continuously against the same store, checking alerting rules (e.g., error rate > 5% for 5 minutes) and, when a rule fires, forwards it to an alertmanager component that handles deduplication, grouping, silencing, and routing to on-call channels. At scale, metrics are sharded across multiple storage instances by service or region, and long-term data is downsampled and moved to cheaper storage while high-resolution recent data stays in fast local storage.

Pull-based scraping keeps the monitoring system simple and lets it detect a target being fully down
A purpose-built time-series database gives efficient compression and fast range/aggregation queries at scale
Separating alert evaluation from dashboards means alerting keeps working even if a dashboard UI is degraded
Downsampling and tiered storage keep long-term retention affordable without losing recent high-resolution detail

AI Mentor Explanation

Designing a metrics system is like a stadium’s groundskeeper walking every section on a fixed schedule to record temperature and moisture readings (metrics), instead of waiting for each section to report in on its own. Each reading is stored with the section’s label so trends per section can be tracked over the match. A separate alert desk continuously checks those readings against thresholds and pages ground staff if any section overheats for several minutes straight. That scheduled-poll, labeled-storage, separate-alerting model is exactly how a metrics monitoring system like Prometheus works.

Step-by-Step Explanation

Step 1
Expose metrics per service
Each service exposes counters, gauges, and histograms on a metrics endpoint.
Step 2
Scrape on an interval
A central scraper polls each endpoint on a fixed interval and writes labeled samples into a time-series database.
Step 3
Query and downsample
A time-series query language aggregates and downsamples series for dashboards, such as rate() over a counter or p99 of a histogram.
Step 4
Evaluate rules and alert
A separate rule evaluator checks alerting rules continuously and forwards firing alerts to an alertmanager for dedup, grouping, and routing.

What Interviewer Expects

Explains pull-based scraping and contrasts it with push, including when a push gateway is needed
Names a time-series database and why it differs from a general-purpose database for this workload
Separates alert rule evaluation from dashboard querying as distinct concerns
Discusses downsampling and tiered retention for long-term storage cost control

Common Mistakes

Using a general-purpose relational database for high-cardinality, high-write time-series data
Coupling alerting directly to the dashboard UI instead of a continuous rule evaluator
Not handling short-lived/batch jobs that a pull-based scraper would miss
Ignoring alert deduplication and grouping, causing alert fatigue from noisy notifications

Best Answer (HR Friendly)

“I would have every service expose simple numeric metrics like request counts and latency, and have a central system poll those on a fixed schedule and store them as time series. On top of that same data, I’d run a separate process that continuously checks alerting rules and pages the on-call engineer, and I’d downsample older data over time so long-term storage stays affordable.”

Code Example

Scrape config and alert rule (Prometheus-style, illustrative)

scrape_configs:
  - job_name: order-service
    scrape_interval: 15s
    static_configs:
      - targets: ["order-service-1:9100", "order-service-2:9100"]

alerting_rules:
  - alert: HighErrorRate
    expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
    for: 5m
    labels:
      severity: page
    annotations:
      summary: "Error rate above 5% for order-service"

Follow-up Questions

How would you handle high-cardinality labels that could overwhelm a time-series database?
When would you use a push gateway instead of pull-based scraping?
How would you downsample old metrics data while keeping recent data at full resolution?
How does the alertmanager deduplicate and group related alerts to avoid paging fatigue?

MCQ Practice

1. Why is pull-based scraping commonly used in metrics monitoring systems?

2. What is the purpose of a push gateway in a pull-based metrics system?

3. Why is alert rule evaluation typically kept separate from the dashboard query path?

Flash Cards

1 / 4

How Would You Design a Metrics Monitoring System (like Prometheus)?

Expected Interview Answer

AI Mentor Explanation

Step-by-Step Explanation

Expose metrics per service

Scrape on an interval

Query and downsample

Evaluate rules and alert