What is remote write and remote read in Prometheus?
Understand Prometheus remote write and remote read: how they stream samples to long-term storage and query historical metrics back for scalable monitoring.
Expected Interview Answer
Remote write and remote read are Prometheus features for integrating with external long-term storage: remote write streams samples out to a remote endpoint as they are ingested, and remote read lets Prometheus query that remote system to fetch historical samples back.
Prometheus local storage is designed for short-to-medium retention, so remote write forwards scraped samples to systems like Thanos, Cortex, Mimir, or VictoriaMetrics for durable, scalable storage. Remote read allows PromQL queries to transparently pull older data from those backends. Remote write batches and buffers samples in a queue, supports relabeling and filtering, and is the more widely used of the two; remote read is less common because many backends provide their own query layers instead.
- Enables long-term retention beyond local disk limits
- Integrates Prometheus with scalable storage backends
- Buffers and retries samples so transient outages don't lose data
- Supports filtering and relabeling of forwarded samples
- Lets central systems aggregate metrics from many Prometheus servers
AI Mentor Explanation
A scorer at a local ground jots every ball into a notebook that only holds one season. To keep history forever, the scorer photocopies each completed page and mails it to a national archive the moment it is filled — that is remote write. When a historian later wants a match from ten years ago, the archive posts the pages back for reading — that is remote read. Prometheus does the same, streaming samples out and querying them back on demand.
Step-by-Step Explanation
Step 1
Understand local storage limits
Prometheus TSDB is built for shorter retention, so long-term history needs an external store.
Step 2
Configure remote_write
Add a remote_write block pointing at the backend's write endpoint, such as Thanos Receive or Mimir.
Step 3
Tune the write queue
Adjust queue_config for capacity, batching, and retries so bursts and outages are handled gracefully.
Step 4
Filter forwarded samples
Use write_relabel_configs to drop or keep specific series before they are sent.
Step 5
Configure remote_read (optional)
Add a remote_read block if you want PromQL to transparently query the remote store for older data.
Step 6
Verify durability and queries
Confirm samples land in the backend and that historical queries return expected data.
What Interviewer Expects
- Clear distinction between remote write and remote read
- Awareness that remote write is far more common than remote read
- Knowledge of backends like Thanos, Cortex, Mimir, VictoriaMetrics
- Understanding of queue_config buffering and retries
- How write_relabel_configs filters forwarded series
Common Mistakes
- Thinking remote write replaces local storage entirely
- Confusing remote read with federation
- Ignoring queue and buffering tuning under high sample volume
- Assuming remote read is required whenever remote write is used
- Forwarding all series without relabel filtering, inflating cost
Best Answer (HR Friendly)
“Remote write lets Prometheus continuously send its collected metrics to an external system that can store them for a long time, while remote read lets it pull that older data back when you query it. Together they let Prometheus keep far more history than its own disk allows.”
Code Example
remote_write:
- url: "https://storage.example.com/api/v1/write"
queue_config:
capacity: 10000
max_shards: 200
batch_send_deadline: 5s
write_relabel_configs:
- source_labels: [__name__]
regex: 'go_.*'
action: drop
remote_read:
- url: "https://storage.example.com/api/v1/read"
read_recent: trueFollow-up Questions
- How is remote write different from Prometheus federation?
- Which backends commonly receive remote write data?
- How does queue_config protect against backend outages?
- Why is remote read less commonly used than remote write?
- How would you reduce the cost of remote write at scale?
MCQ Practice
1. What does Prometheus remote write do?
Remote write forwards samples to an external write endpoint as they are ingested.
2. Which config section filters series before they are forwarded via remote write?
write_relabel_configs applies relabeling and dropping to samples on the remote write path.
3. Why is remote read less common than remote write?
Many long-term storage systems offer their own query interfaces, so remote read is used less than remote write.
Flash Cards
Remote write — Streams ingested samples out to an external storage endpoint as they arrive.
Remote read — Lets Prometheus query a remote store to fetch historical samples back into PromQL.
Common remote write backends — Thanos, Cortex, Mimir, VictoriaMetrics and similar long-term stores.
queue_config — Controls buffering, sharding, batching and retries on the remote write path.
write_relabel_configs — Filters or drops series before they are forwarded via remote write.
Continue Learning
Related Interview Questions
How do you handle long-term storage for Prometheus metrics (Thanos, Cortex, Mimir)?
hard
What is cardinality in Prometheus and why is high cardinality a problem?
medium
What is Prometheus and what problems does it solve in monitoring?
easy
What is the difference between push and pull metric collection?
medium