What is the difference between Kafka retention by time and by size?
Learn the difference between Kafka retention by time (retention.ms) and by size (retention.bytes), how they combine, and when each is used, with examples.
Expected Interview Answer
Kafka retention decides how long messages stay in a topic before deletion: time-based retention (retention.ms) removes segments once they are older than a configured age, while size-based retention (retention.bytes) removes the oldest segments once a partition exceeds a configured byte budget.
Both policies operate per partition on the log's closed segments, and Kafka applies whichever limit is hit first — so a topic can be configured with both, and a message is eligible for deletion as soon as either the age or the size threshold is crossed. Time retention gives predictable data-availability windows (for example, keep seven days for replay), while size retention protects disk from unbounded growth regardless of traffic bursts. Neither policy applies to compacted topics, where cleanup.policy=compact keeps the latest value per key instead of deleting by age or size.
- Time retention guarantees a predictable replay window
- Size retention caps disk usage during traffic spikes
- Both can be combined for bounded age and bounded storage
- Configurable per topic, overriding broker defaults
- Deletion happens at segment granularity, keeping it cheap
AI Mentor Explanation
Think of a dressing-room notice board that pins up match reports. A time policy says any report older than a week comes down automatically, whether it is one sheet or fifty. A size policy says the board can only hold twenty sheets, so the oldest is unpinned when a twenty-first arrives. Kafka keeps messages exactly like this: expire by age, or drop the oldest once the space budget is full, whichever limit is reached first.
Step-by-Step Explanation
Step 1
Set the time budget
Configure retention.ms (or retention.minutes/hours) on the topic to define the maximum age a message may reach before its segment becomes eligible for deletion.
Step 2
Set the size budget
Configure retention.bytes to cap the total stored bytes per partition; the oldest segments are removed once the partition exceeds this ceiling.
Step 3
Understand segment granularity
Retention acts on closed log segments, not individual messages, so cleanup happens when a whole segment ages out or pushes the partition past its size limit.
Step 4
Combine both when needed
Set both properties together; Kafka deletes as soon as either the age or the size threshold is hit, giving bounded age and bounded storage simultaneously.
Step 5
Exclude compacted topics
Remember that cleanup.policy=compact keeps the latest value per key instead of deleting by time or size, so retention limits behave differently there.
What Interviewer Expects
- Knowing retention.ms controls age and retention.bytes controls size
- Understanding that whichever limit is hit first triggers deletion
- Awareness that retention is per partition and acts on segments
- Distinguishing delete retention from log compaction
- Choosing the right policy for replay windows versus disk safety
Common Mistakes
- Thinking retention.bytes applies to the whole topic rather than per partition
- Assuming a message is deleted the instant it ages out, not when its segment closes
- Confusing delete retention with compaction
- Believing only one of time or size can be set at a time
- Ignoring that low segment.ms is needed for retention to act promptly
Best Answer (HR Friendly)
“Kafka can decide when to throw away old messages either by how old they are or by how much disk space they take up. Time-based retention keeps data for a set period like seven days, while size-based retention keeps only a set amount of data and drops the oldest when that fills up.”
Code Example
# Keep messages for 7 days
kafka-configs.sh --bootstrap-server localhost:9092 \
--alter --entity-type topics --entity-name orders \
--add-config retention.ms=604800000
# Also cap each partition at 1 GB (whichever limit hits first wins)
kafka-configs.sh --bootstrap-server localhost:9092 \
--alter --entity-type topics --entity-name orders \
--add-config retention.bytes=1073741824Follow-up Questions
- What role does segment.ms and segment.bytes play in when retention actually deletes data?
- How does log compaction differ from delete-based retention?
- Can you set retention.bytes to -1, and what does that mean?
- How would you keep unlimited data for replay while protecting disk?
- What happens to a consumer that is slower than the retention window?
MCQ Practice
1. If a topic sets both retention.ms and retention.bytes, when is a segment deleted?
Kafka evicts the oldest segment as soon as either the age or the byte threshold is crossed — whichever comes first.
2. retention.bytes applies to which scope?
retention.bytes is enforced per partition, so a topic with many partitions can hold far more than the single value implies.
3. Which cleanup policy ignores time and size retention in favor of keeping the latest value per key?
cleanup.policy=compact retains the most recent record for each key instead of deleting purely by age or size.
Flash Cards
What does retention.ms control? — The maximum age a message may reach before its segment becomes eligible for deletion.
What does retention.bytes control? — The maximum stored bytes per partition; the oldest segments are removed once exceeded.
With both set, which wins? — Whichever limit — age or size — is reached first triggers deletion.
Does retention apply to compacted topics? — No; compaction keeps the latest value per key rather than deleting by time or size.