Time- and Size-Based Retention
Kafka's default retention policy deletes data based on age and/or total size, controlled by broker-level defaults log.retention.hours (168, i.e., 7 days), log.retention.bytes (-1, unlimited, per partition), and log.retention.check.interval.ms (5 minutes) which governs how often the retention thread scans for eligible data. Retention is not about individual messages expiring instantly the moment they age past the limit — it operates on whole log segments, so a message can technically live slightly longer than the configured retention until its containing segment is fully eligible for removal.
Cricket analogy: A cricket stadium's CCTV footage is auto-deleted after 7 days by policy, but if a recording file spans slightly past midnight on day 7, the whole file is kept until it fully clears the retention window, like Kafka deleting whole segments rather than exact timestamps.
How Segments Enable Efficient Deletion
Each partition's log is physically split into segment files (e.g., 00000000000000000000.log), rolled over once a segment reaches log.segment.bytes (default 1GB) or log.roll.ms, whichever comes first. Retention deletion only ever removes entire closed segments whose newest message exceeds the retention window — never the active segment currently being written to — which is why Kafka can delete gigabytes of old data cheaply via a filesystem unlink rather than rewriting logs record by record, unlike compaction which rewrites segments.
Cricket analogy: A cricket scorebook fills one page per innings; once a page is full it's closed and archived, and only fully archived pages can be discarded later, while the current page being written is never touched, like Kafka never deleting the active segment.
Per-Topic Overrides
Retention settings configured at the broker level (server.properties) act only as defaults; every topic can override them individually using kafka-configs.sh --alter --entity-type topics --entity-name orders --add-config retention.ms=259200000, which is essential when different topics have very different data lifecycles — a clickstream topic might need only 24 hours of retention while a financial audit topic needs 90 days or longer.
Cricket analogy: A cricket board sets a league-wide 5-year archive policy for match footage, but grants Test matches at iconic venues like the MCG a 25-year exception, similar to a topic overriding the broker's default retention.ms.
# Override retention for a specific topic to 3 days
kafka-configs.sh --bootstrap-server broker1:9092 \
--alter --entity-type topics --entity-name clickstream-events \
--add-config retention.ms=259200000,retention.bytes=536870912000
# Check current effective retention for a topic
kafka-configs.sh --bootstrap-server broker1:9092 \
--describe --entity-type topics --entity-name clickstream-eventsRetention vs Consumer Offsets
Retention deletion is completely independent of consumer progress — Kafka does not check whether any consumer group has actually read a message before deleting it once retention expires. A consumer group that falls behind (high consumer lag) risks permanently missing messages that age out before it catches up, which is why monitoring consumer lag against retention windows, e.g., via kafka-consumer-groups.sh --describe, is critical for at-least-once processing guarantees.
Cricket analogy: A stadium's ticket-scan logs get purged after 30 days regardless of whether the ticketing office has finished auditing them, so an auditor who falls behind schedule risks losing records before finishing the review, like a lagging Kafka consumer missing deleted data.
Retention is a hard deletion with no soft-delete or recycle bin — once a segment ages out and is removed, the data is gone. A consumer group that has been down for longer than the topic's retention.ms will silently lose the messages produced during that gap; always alert on consumer lag approaching the retention window.
- Default retention is governed by log.retention.hours/ms and log.retention.bytes at the broker level.
- Retention deletes whole closed log segments, never the currently active segment being written.
- Segments roll over based on log.segment.bytes or log.roll.ms, whichever comes first.
- Every topic can override broker retention defaults individually via kafka-configs.sh.
- Retention deletion is completely independent of consumer offsets.
- Monitoring consumer lag against a topic's retention window is essential to avoid silent data loss.
Practice what you learned
1. What is the default value of log.retention.hours in Kafka?
2. What does Kafka's retention deletion actually operate on?
3. How can you set a shorter retention for one specific topic without changing the broker default?
4. What happens if a consumer group's lag exceeds a topic's retention window?
5. Which config controls the maximum size of an individual log segment file?
Was this page helpful?
You May Also Like
Log Compaction
How Kafka's compaction cleanup policy retains only the latest value per key instead of expiring records by time.
Kafka Replication and Leaders
How Kafka replicates partitions across brokers and elects leaders to guarantee durability and availability.
The Kafka Controller
The broker role responsible for cluster-wide coordination: leader election, metadata propagation, and admin operations.
Related Reading
Related Study Notes in DevOps
Browse all study notesNginx Study Notes
DevOps · 30 topics
DevOpsAnsible Study Notes
DevOps · 30 topics
DevOpsAdvanced Kubernetes Study Notes
Kubernetes · 30 topics
DevOpsAdvanced Bash Scripting Study Notes
Bash · 30 topics
DevOpsDocker & Kubernetes Study Notes
YAML · 40 topics
DevOpsCI/CD Tools & Pipelines Study Notes
YAML · 37 topics