How do you handle long-term storage for Prometheus metrics (Thanos, Cortex, Mimir)?
How Thanos, Cortex, and Mimir give Prometheus durable long-term storage via object storage, deduplication, compaction, and downsampling for years of metrics.
Expected Interview Answer
Prometheus stores data locally for a limited retention window, so long-term storage is handled by pairing it with a system like Thanos, Cortex, or Mimir that ships or receives metrics into cheap object storage (S3, GCS, Azure Blob) and serves unified, durable, queryable history.
Thanos runs a sidecar next to each Prometheus that uploads TSDB blocks to object storage, then a Store Gateway, Querier, and Compactor provide global query and downsampling. Cortex and Mimir take a push model: Prometheus remote-writes samples into a horizontally sharded, multi-tenant cluster that also persists blocks to object storage. All three decouple retention from local disk, add high availability by deduplicating samples from replicated Prometheus pairs, and use compaction plus downsampling to keep years of data cheap to store and fast to query.
- Retention measured in years instead of days
- Cheap, durable object storage for blocks
- Global query view across many Prometheus servers
- High availability via sample deduplication
- Downsampling keeps long-range queries fast
- Multi-tenancy for shared platforms (Cortex/Mimir)
AI Mentor Explanation
A single scorer at the ground can only keep this season's books in the pavilion cupboard before it overflows. Long-term storage is like couriering each completed scorebook to a national cricket archive, where clerks index every match so anyone can pull up a batter's record from twenty years ago in seconds, even though no ground keeps that much paper on site.
Step-by-Step Explanation
Step 1
Identify the retention limit
Recognise that a single Prometheus keeps data on local disk for a bounded window (commonly 15 days), which is unsuitable for compliance or long-range trends.
Step 2
Choose a shipping model
Pick sidecar upload (Thanos reads TSDB blocks and uploads to object storage) or remote-write push (Cortex/Mimir receive samples over the remote_write protocol).
Step 3
Persist to object storage
Configure an S3/GCS/Azure Blob bucket as the durable backend where TSDB blocks live cheaply and indefinitely.
Step 4
Add global query and HA
Deploy a Querier/Query-Frontend that fans out across all sources and deduplicates samples from replicated Prometheus pairs so one dataset is returned.
Step 5
Compact and downsample
Run a Compactor to merge blocks and produce 5m and 1h downsampled resolutions, keeping long-range queries fast and storage costs low.
Step 6
Enforce retention and tenancy
Set per-resolution retention policies and, for shared platforms, isolate tenants so teams share the cluster without seeing each other's metrics.
What Interviewer Expects
- Why local Prometheus retention is insufficient
- Difference between the Thanos sidecar model and Cortex/Mimir remote-write model
- Role of object storage as the durable backend
- How sample deduplication provides high availability
- What compaction and downsampling achieve
- Awareness of multi-tenancy in Mimir/Cortex
Common Mistakes
- Claiming Prometheus alone can store years of data reliably
- Confusing remote_write (push) with the Thanos sidecar (upload) model
- Ignoring the need for deduplication when running HA Prometheus pairs
- Forgetting that downsampling is what keeps long-range queries fast
- Assuming object storage is optional rather than the core of these systems
Best Answer (HR Friendly)
“Prometheus only keeps recent data on its own disk, so we add a system like Thanos, Cortex, or Mimir that copies the metrics into cheap cloud storage. That lets us keep years of history, search across all our servers at once, and still have everything stay fast and available.”
Code Example
remote_write:
- url: http://mimir:9009/api/v1/push
headers:
X-Scope-OrgID: team-a
queue_config:
max_samples_per_send: 2000
capacity: 10000
# Thanos sidecar alternative (run beside Prometheus):
# thanos sidecar \
# --tsdb.path=/prometheus \
# --objstore.config-file=bucket.yaml \
# --prometheus.url=http://localhost:9090Follow-up Questions
- How does the Thanos Compactor's downsampling improve query performance?
- How do Thanos and Mimir deduplicate samples from HA Prometheus pairs?
- What are the trade-offs between the sidecar and remote-write approaches?
- How would you configure multi-tenancy in Grafana Mimir?
- How does object storage cost compare to keeping data on local SSD?
MCQ Practice
1. Which component uploads Prometheus TSDB blocks to object storage in a Thanos deployment?
The Thanos sidecar runs next to Prometheus, reads its TSDB blocks, and uploads them to object storage for long-term retention.
2. How do Cortex and Mimir primarily ingest metrics from Prometheus?
Cortex and Mimir use a push model where Prometheus sends samples to them over the remote_write protocol.
3. What is the main purpose of downsampling in long-term Prometheus storage?
Downsampling produces lower-resolution copies (e.g. 5m, 1h) so queries over months or years stay fast and storage stays affordable.
Flash Cards
Why can't Prometheus alone handle long-term storage? — It keeps data on local disk for a bounded retention window, with no built-in durable or global backend.
Thanos vs Cortex/Mimir ingestion model? — Thanos uploads TSDB blocks via a sidecar; Cortex/Mimir receive samples via remote_write push.
What backend do all three use for durability? — Object storage — S3, GCS, or Azure Blob — where TSDB blocks live cheaply and indefinitely.
What provides HA across replicated Prometheus pairs? — Sample deduplication, so identical series from both replicas return as one dataset.
What does the Compactor do? — Merges blocks and creates downsampled resolutions to speed up long-range queries and cut cost.
Continue Learning
Related Interview Questions
What is remote write and remote read in Prometheus?
medium
A single Prometheus can no longer scrape every target. How do you shard the scrape load?
hard
What are common Prometheus monitoring best practices and pitfalls?
medium
How do you run Prometheus and Alertmanager in high availability without getting duplicate pages?
hard