How does Prometheus store time-series data (the TSDB)?
Learn how the Prometheus TSDB stores metrics: head block, write-ahead log, immutable disk blocks, compaction, and retention explained for interviews.
Expected Interview Answer
Prometheus stores metrics in a purpose-built local time-series database (TSDB) that appends samples to an in-memory head block backed by a write-ahead log, then periodically compacts them into immutable on-disk blocks.
Each series is identified by its metric name plus label set and holds a stream of timestamp-value samples. Recent data lives in the head block in memory, protected against crashes by a write-ahead log (WAL). Every two hours the head is flushed into a persistent block on disk containing compressed chunks, an index, and metadata; a background compactor merges these smaller blocks into larger ones. Old blocks beyond the retention window are deleted, and querying works by selecting series from the index and reading the relevant chunks.
- Highly efficient compression of timestamp-value chunks
- Crash safety through the write-ahead log
- Fast label-based lookups via an inverted index
- Immutable blocks simplify compaction and retention
- Local storage means no external database dependency
AI Mentor Explanation
The TSDB is like scoring a match on a live worksheet, then binding completed sessions into permanent volumes. Ball-by-ball runs go onto the current sheet (the head block) while a carbon copy is kept in case ink smudges (the WAL). At each interval the finished sheet is bound into a sealed ledger (a disk block), and old ledgers past the archive limit are discarded, exactly how Prometheus flushes the head and prunes blocks beyond retention.
Step-by-Step Explanation
Step 1
Ingest samples
Scraped timestamp-value samples for each series (metric name + labels) are appended to the in-memory head block.
Step 2
Journal to the WAL
Every append is also written to the write-ahead log so unflushed data survives a crash and can be replayed on restart.
Step 3
Cut a block
About every two hours the head is flushed to a persistent, immutable block containing compressed chunks, an index, and metadata.
Step 4
Compact blocks
A background compactor merges adjacent smaller blocks into larger ones, reducing overhead and improving query efficiency.
Step 5
Enforce retention
Blocks older than the configured retention time or size are deleted, and queries read chunks selected via the block index.
What Interviewer Expects
- Understanding of series identity as metric name plus label set
- The role of the head block and write-ahead log
- How blocks are cut roughly every two hours and made immutable
- The purpose of compaction and the inverted index
- How retention deletes old blocks by time or size
Common Mistakes
- Believing Prometheus stores data in an external SQL database by default
- Confusing the WAL with long-term storage
- Thinking blocks are mutable and updated in place
- Ignoring that high label cardinality explodes the number of series
- Assuming remote_write replaces local TSDB rather than supplementing it
Best Answer (HR Friendly)
“Prometheus keeps its metrics in its own local database designed for time-based data. New readings are held in memory and written to a safety log, then packed into compressed files on disk every couple of hours, and anything older than the retention limit is automatically deleted.”
Code Example
# Start Prometheus with retention flags
# prometheus --storage.tsdb.path=/data \
# --storage.tsdb.retention.time=15d \
# --storage.tsdb.retention.size=50GB
# On-disk layout of the data directory:
# /data
# |- wal/ # write-ahead log for the head block
# |- chunks_head/ # memory-mapped head chunks
# |- 01HRT-ABC/ # a persisted, immutable block
# | |- chunks/ # compressed timestamp-value chunks
# | |- index # inverted index of series by label
# | |- meta.json # block time range and stats
# |- 01HRT-XYZ/ # another block, later compactedFollow-up Questions
- What is the write-ahead log and why does it matter?
- How does high label cardinality affect the TSDB?
- What is the difference between local storage and remote_write?
- How does compaction improve query performance?
- How do you configure retention by time versus by size?
MCQ Practice
1. What protects unflushed head-block data against a crash?
The WAL records every append so recent, unflushed samples can be replayed and recovered after a restart.
2. How often does Prometheus typically cut the head into a persistent block?
By default the head block is flushed to an immutable on-disk block roughly every two hours, then compacted over time.
3. What uniquely identifies a Prometheus time series?
A series is defined by its metric name combined with the exact set of label key-value pairs; changing any label creates a new series.
Flash Cards
What is the head block? — The in-memory block holding the most recent samples before they are flushed to disk.
What is the WAL? — The write-ahead log that journals appends so unflushed data survives a crash and can be replayed.
How often is a block cut? — About every two hours the head is flushed into an immutable on-disk block.
What identifies a series? — Its metric name plus its complete label set; any label change creates a new series.
How is old data removed? — Blocks past the configured retention time or size are deleted automatically.