How Would You Design a Distributed Logging System?
System design guide to distributed logging: local buffering, Kafka ingestion, indexing, retention, and trace correlation.
Expected Interview Answer
Design a distributed logging system as a pipeline: lightweight agents on each host tail and buffer log lines locally, ship them to a durable, partitioned ingestion queue like Kafka, and stream consumers index the logs into a searchable store while applying retention and sampling policies to keep cost and volume manageable.
Each service instance writes logs to local disk (or stdout, captured by a sidecar/agent), and a lightweight forwarder tails these files, batches lines, and pushes them asynchronously to a durable ingestion layer such as Kafka, partitioned by service or log source so ingestion scales horizontally. This buffering means the application itself never blocks on log delivery, and the agent can retry without losing lines if the ingestion layer is briefly unavailable. Downstream consumers read from the queue and write into an indexed store (e.g., Elasticsearch or a columnar log store) optimized for full-text and structured queries, tagging each entry with metadata like service name, host, trace ID, and severity. To control cost at scale, the system applies tiered retention (hot index for recent logs, cold cheap storage like object storage for older logs) and sampling or downsampling for high-volume debug-level logs, while always retaining error/warn logs at full fidelity. Correlating logs across services for a single request is done via a trace ID propagated through headers, so an engineer can pull every log line for one request across every microservice it touched.
- Local buffering and async shipping decouple application performance from logging pipeline health
- Partitioned ingestion via a durable queue absorbs traffic spikes without losing log lines
- Tiered retention and sampling keep storage and query cost bounded at high log volume
- Trace ID propagation lets engineers reconstruct a single request's path across many services
AI Mentor Explanation
Designing distributed logging is like every fielder keeping a personal notebook of what happened on each ball instead of shouting it to the scorer live, then handing batches of notes to a runner who carries them to the official scorebook. If the runner is briefly delayed, notebooks just fill up a bit more instead of losing any entries. The scorebook indexes entries by over and player so anyone can search what happened on ball twelve of the ninth over instantly. That local-buffer, batched-hand-off, and indexed-archive model is exactly how distributed logging pipelines work.
Step-by-Step Explanation
Step 1
Write logs locally
Each service instance writes to local disk or stdout without blocking on network delivery.
Step 2
Buffer and batch-ship
A lightweight agent tails logs, batches lines, and pushes them asynchronously to a durable, partitioned queue like Kafka.
Step 3
Consume and index
Stream consumers read from the queue and write into a searchable, indexed store tagged with service, host, trace ID, and severity.
Step 4
Apply retention and sampling
Tiered retention moves old logs to cheap cold storage; high-volume debug logs are sampled while errors are kept at full fidelity.
What Interviewer Expects
- Explains local buffering so the application never blocks on log delivery
- Names a durable, partitioned ingestion layer (e.g., Kafka) between producers and indexers
- Discusses tiered retention/sampling to control cost at high log volume
- Mentions trace ID propagation for correlating logs across services for one request
Common Mistakes
- Having the application write logs synchronously to a remote service, risking blocking on log failures
- No durable buffer, so log lines are lost when the indexing store is briefly down
- Ignoring cost: indexing every debug log at full fidelity forever
- Forgetting a correlation ID, making cross-service debugging nearly impossible
Best Answer (HR Friendly)
“I would have each service write logs locally first so nothing ever blocks waiting on the logging system, then ship those logs in batches to a durable queue that feeds a searchable index. To keep costs under control I’d keep recent and error logs fully searchable but move older, low-priority logs to cheaper storage, and I’d tag every log with a request ID so we can trace one request across every service it touched.”
Code Example
agent:
tail: /var/log/app/*.log
batchSizeLines: 500
batchIntervalMs: 2000
sink: kafka://logs-ingest
partitionKey: serviceName
retention:
tiers:
- name: hot
store: elasticsearch
maxAgeDays: 7
- name: cold
store: s3
maxAgeDays: 365
sampling:
debug: 0.05 # keep 5% of debug lines
info: 0.5 # keep 50% of info lines
warn: 1.0
error: 1.0Follow-up Questions
- How would you propagate and use a trace ID to correlate logs across microservices?
- How would you prevent a single noisy service from overwhelming the shared ingestion queue?
- What is the trade-off between structured logging (JSON) and plain text logs at scale?
- How would you alert on log patterns (e.g., error rate spikes) without scanning every log line manually?
MCQ Practice
1. Why do distributed logging agents buffer logs locally before shipping them?
Local buffering decouples the app from downstream health, and batches can be retried without dropping lines.
2. What is the purpose of tiered retention in a logging system?
Tiered retention balances query performance and cost by aging logs from expensive hot storage to cheap cold storage.
3. Why propagate a trace ID through service calls in a logging system?
A trace ID tags every log line generated while handling one request, letting engineers filter and follow it across services.
Flash Cards
Why buffer logs locally before shipping? — So the application never blocks on log delivery and lines survive brief downstream outages.
What role does a durable queue like Kafka play? — It absorbs bursts and decouples log producers from indexing consumers.
Why use tiered retention? — To keep recent/error logs fast to search while moving old low-priority logs to cheap storage.
Why propagate a trace ID? — To correlate every log line for a single request across all the services it touched.