What are common Prometheus monitoring best practices and pitfalls?
Prometheus best practices: metric naming, bounded labels, recording rules, and symptom-based alerting, while avoiding cardinality explosions and pitfalls.
Expected Interview Answer
Best practices include following naming and label conventions, avoiding high-cardinality labels, exposing metrics via a pull-based exporter model, recording expensive queries, and alerting on symptoms; the common pitfalls are cardinality explosions, using labels for unbounded values, over-alerting, and treating Prometheus as long-term or event storage.
Well-run setups name metrics with a unit suffix (e.g. http_request_duration_seconds), keep labels to bounded, low-cardinality dimensions, and never put user IDs, emails, or full URLs in labels. They use recording rules to precompute costly aggregations, apply the RED/USE methods to decide what to measure, and pair Prometheus with Alertmanager for grouping and routing. The classic pitfalls are cardinality explosions that blow up memory, scraping too frequently, relying on Prometheus for years of retention without Thanos/Mimir, and building brittle dashboards on ad-hoc queries instead of recording rules.
- Predictable memory and stable performance
- Consistent, discoverable metric names
- Fast dashboards via recording rules
- Actionable, symptom-based alerts
- Clear measurement strategy with RED/USE
- Avoids costly cardinality and retention mistakes
AI Mentor Explanation
A good scorer uses standard column headings every match so any analyst can read the book instantly, and never invents a fresh column for each spectator. Prometheus best practice is the same: consistent metric names and bounded labels. Add a unique label per user and your scorebook grows a new column for every fan until it collapses under its own weight.
Step-by-Step Explanation
Step 1
Name metrics consistently
Use base units and a suffix like _seconds or _bytes, with a clear namespace, so metrics are self-describing and discoverable.
Step 2
Keep labels bounded
Only use labels whose values come from a small, fixed set; never store user IDs, emails, or full URLs as label values.
Step 3
Choose what to measure
Apply RED (Rate, Errors, Duration) for services and USE (Utilization, Saturation, Errors) for resources to pick meaningful signals.
Step 4
Precompute with recording rules
Move expensive aggregations into recording rules so dashboards query cheap precomputed series instead of heavy ad-hoc PromQL.
Step 5
Alert on symptoms
Write SLO-based alerts on user impact and route them through Alertmanager with grouping and severity tiers.
Step 6
Plan retention separately
Treat Prometheus as short-term storage and add Thanos, Cortex, or Mimir for durable long-term history.
What Interviewer Expects
- Metric naming and unit-suffix conventions
- Strong awareness of label cardinality and its dangers
- Knowledge of RED and USE methods
- Use of recording rules for expensive queries
- Symptom-based alerting through Alertmanager
- Understanding that Prometheus is not long-term or event storage
Common Mistakes
- Putting unbounded values (user IDs, URLs) in labels, causing cardinality explosions
- Scraping far too frequently and overloading targets
- Relying on Prometheus alone for years of retention
- Building dashboards on ad-hoc queries instead of recording rules
- Over-alerting on every metric rather than symptoms
- Using Prometheus for high-precision event or log storage
Best Answer (HR Friendly)
“Good Prometheus practice means naming metrics clearly, keeping labels simple, and precomputing heavy queries so dashboards stay fast. The biggest trap is putting unique values like user IDs into labels, which creates a memory-crushing explosion of data — and expecting Prometheus to store years of history on its own.”
Code Example
groups:
- name: recording-rules
rules:
- record: job:http_request_duration_seconds:p99
expr: |
histogram_quantile(0.99,
sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))# BAD: unbounded label -> cardinality explosion
http_requests_total{user_id="48213", url="/orders/9f3a"} 1
# GOOD: bounded labels only
http_requests_total{method="GET", route="/orders/:id", status="200"} 1Follow-up Questions
- What causes a cardinality explosion and how do you detect it?
- When should you use a recording rule versus a raw query?
- What are the RED and USE methods for choosing metrics?
- Why is Prometheus a poor fit for event or log storage?
- How do you choose an appropriate scrape interval?
MCQ Practice
1. Which practice most directly causes a Prometheus cardinality explosion?
Unbounded label values like user IDs create a new time series per value, multiplying series count and exhausting memory.
2. What is the main purpose of a Prometheus recording rule?
Recording rules precompute costly aggregations so dashboards and alerts read cheap precomputed series instead of heavy ad-hoc PromQL.
3. Which method helps decide what to measure for a service?
The RED method — Rate, Errors, Duration — is a standard way to pick the key signals to monitor for a request-serving service.
Flash Cards
Metric naming convention? — Use base units with a suffix like _seconds or _bytes and a clear namespace so metrics are self-describing.
What is a cardinality explosion? — Putting unbounded values in labels creates a new series per value, multiplying series count and exhausting memory.
What are the RED and USE methods? — RED: Rate, Errors, Duration for services. USE: Utilization, Saturation, Errors for resources.
When to use a recording rule? — When a query is expensive or reused; precompute it into a new series for fast dashboards and alerts.
Is Prometheus long-term storage? — No — it's short-term; add Thanos, Cortex, or Mimir for durable multi-year retention.
Continue Learning
Related Interview Questions
What are labels in Prometheus and why are they powerful?
easy
What is cardinality in Prometheus and why is high cardinality a problem?
medium
What are the RED and USE methods for monitoring?
medium
How does PromQL vector matching work, and how do you enrich a metric with labels from an info metric using group_left?
hard