What is cardinality in Prometheus and why is high cardinality a problem?
Understand cardinality in Prometheus, why high cardinality drains memory and slows queries, which labels cause it, and how to detect and control the problem.
Expected Interview Answer
Cardinality is the number of unique time series Prometheus stores, driven by every distinct combination of a metric name and its label values. High cardinality means an explosion of series, which consumes memory and CPU, slows queries, and can crash the server.
Each unique label-value combination creates its own time series that Prometheus must track in memory and index. Adding a single label with many possible values multiplies the series count, and putting unbounded values like user IDs, email addresses, or request IDs into labels can produce millions of series. This inflates the head block and index, drives up RAM usage, and makes queries and compaction expensive, sometimes leading to out-of-memory failures.
- Understanding it keeps memory usage predictable
- Prevents server out-of-memory crashes
- Keeps PromQL queries fast
- Guides safe label design
- Improves long-term storage and compaction
AI Mentor Explanation
Imagine giving every single ball its own uniquely numbered scorebook page instead of tallying runs per over. With a handful of overs that is fine, but tag each delivery with an exact millisecond timestamp and you suddenly need a fresh page for every ball ever bowled, and the scorer drowns under an unmanageable mountain of near-identical pages nobody can search.
Step-by-Step Explanation
Step 1
Define cardinality
Count the total unique time series, which equals the product of distinct values across labels for each metric.
Step 2
Spot risky labels
Flag labels whose values are unbounded — user IDs, emails, request IDs, timestamps, full URLs.
Step 3
Measure current series
Use prometheus_tsdb_head_series and topk on series counts to find the worst offenders.
Step 4
Reduce dimensions
Drop or bucket high-cardinality labels via relabeling, or aggregate before ingestion.
Step 5
Set guardrails
Apply sample_limit and label_limit in scrape configs to cap runaway series at the source.
What Interviewer Expects
- Cardinality defined as the count of unique time series
- How label-value combinations multiply series
- The memory and query cost of high cardinality
- Examples of labels that cause explosions
- Mitigation strategies like relabeling and limits
Common Mistakes
- Thinking cardinality is about the number of metrics, not series
- Adding user IDs, request IDs, or timestamps as labels
- Ignoring that each label multiplies existing series
- Not monitoring head series growth until the server crashes
Best Answer (HR Friendly)
“Cardinality is how many distinct data streams Prometheus has to keep track of. If you tag your data with something that has endless possible values, like a unique ID per user, the number of streams explodes and can overwhelm the server's memory and slow everything down.”
Code Example
# Total series currently in the head block
prometheus_tsdb_head_series
# Top 10 metric names by number of series
topk(10, count by (__name__)({__name__=~".+"}))Follow-up Questions
- How do you find which metric is causing a cardinality explosion?
- How can relabeling reduce cardinality?
- What are sample_limit and label_limit in scrape configs?
- Why do unbounded label values crash Prometheus?
- How does cardinality affect long-term storage like Thanos or Cortex?
MCQ Practice
1. What does cardinality measure in Prometheus?
Cardinality is the count of unique time series, determined by metric name and label-value combinations.
2. Which label is most likely to cause a cardinality explosion?
request_id is unbounded, so each request creates a new series, rapidly multiplying cardinality.
3. Which setting caps series per scrape at the source?
sample_limit rejects a scrape if it exposes more than the configured number of samples, guarding against runaway cardinality.
Flash Cards
What is cardinality? — The number of unique time series Prometheus stores, from metric name plus label-value combinations.
Why is high cardinality bad? — It consumes memory and CPU, slows queries, and can crash the server with out-of-memory errors.
Name a dangerous label value. — Unbounded values like user IDs, request IDs, emails, or raw timestamps.
How do you cap series at scrape time? — Use sample_limit and label_limit in the scrape config.