How does log compaction work in Kafka?
Learn how Kafka log compaction keeps the latest value per key, how tombstones delete keys, and why it powers changelog and state-store topics.
Expected Interview Answer
Log compaction is a Kafka retention policy that keeps only the most recent value for each message key within a topic, discarding older records with the same key so the log retains the latest state of every key indefinitely.
Unlike time or size based retention that deletes whole segments regardless of content, compaction guarantees that at least the last update for each key survives, making it ideal for changelog and state topics. A background cleaner thread scans the log, builds an offset map of the latest offset per key, and rewrites segments dropping superseded records. Records with a null value act as tombstones, marking a key for deletion, and are themselves removed after delete.retention.ms. Compacted topics still keep a recent 'head' of the log uncompacted so consumers can read every event that arrived within the compaction lag window.
- Retains the latest state per key without unbounded growth
- Enables topics to act as durable key-value changelogs
- Supports fast state restoration for stream processors
- Allows deletion of keys via tombstone records
- Keeps storage bounded by number of distinct keys, not total events
AI Mentor Explanation
Think of log compaction as maintaining a batting average sheet rather than every ball-by-ball entry. Each time a player's average updates you only need the newest figure, not the entire history of past averages. Compaction sweeps the record and keeps just the latest average per player, throwing away the superseded numbers, so the sheet always shows the current state of every batter without ballooning in size.
Step-by-Step Explanation
Step 1
Enable compaction
Set cleanup.policy=compact on the topic so Kafka retains the latest value per key instead of deleting by time or size.
Step 2
Log split into head and tail
The recent 'head' stays uncompacted so consumers see every event; the older 'tail' is eligible for cleaning.
Step 3
Cleaner builds offset map
A background log-cleaner thread scans the tail and records the latest offset for each key in an in-memory map.
Step 4
Rewrite segments
The cleaner copies live records forward, dropping any record whose key has a newer offset, and replaces old segments.
Step 5
Handle tombstones
Records with a null value mark a key for deletion and are retained for delete.retention.ms before being fully removed.
What Interviewer Expects
- Definition of compaction as keeping the latest value per key
- Contrast with time/size-based deletion retention
- Understanding of tombstone (null-value) records for deletion
- Awareness of the log-cleaner thread and offset map
- Use cases like changelog and state-store topics
Common Mistakes
- Thinking compaction guarantees every intermediate value is kept
- Confusing cleanup.policy=compact with cleanup.policy=delete
- Forgetting that null-value records are tombstones that delete keys
- Assuming the entire log is always compacted, ignoring the uncompacted head
Best Answer (HR Friendly)
“Log compaction keeps only the newest version of each keyed message and throws away the older ones, so a Kafka topic can hold the current state of everything without growing forever. It is like a contact list that keeps just each person's latest phone number instead of every number they ever had.”
Code Example
# Create a compacted topic for a user-profile changelog
kafka-topics.sh --bootstrap-server localhost:9092 \
--create --topic user-profiles \
--partitions 6 --replication-factor 3 \
--config cleanup.policy=compact \
--config min.cleanable.dirty.ratio=0.5 \
--config delete.retention.ms=86400000
# Produce a tombstone to delete a key (null value)
kafka-console-producer.sh --bootstrap-server localhost:9092 \
--topic user-profiles --property parse.key=true \
--property key.separator=: <<< 'user-7:'Follow-up Questions
- What is a tombstone record and how long is it retained?
- How does min.cleanable.dirty.ratio affect when compaction runs?
- Can a topic use both compaction and time-based deletion together?
- Why are compacted topics ideal for Kafka Streams state stores?
- What is the difference between the log head and log tail during compaction?
MCQ Practice
1. What does log compaction guarantee to retain?
Compaction keeps at least the most recent value per key, discarding older superseded records with the same key.
2. What is a record with a null value in a compacted topic?
A null-value record is a tombstone; it signals the key should be deleted and is retained for delete.retention.ms.
3. Which cleanup.policy enables log compaction?
Setting cleanup.policy=compact tells Kafka to retain the latest value per key instead of deleting by time or size.
Flash Cards
What is Kafka log compaction? — A retention policy that keeps only the latest value for each message key, discarding older records with that key.
What is a tombstone? — A record with a null value that marks a key for deletion; retained for delete.retention.ms then removed.
How does compaction differ from delete retention? — Delete removes whole segments by time/size; compaction keeps the newest value per key regardless of age.
What does the log cleaner do? — A background thread that builds an offset map of the latest offset per key and rewrites segments dropping superseded records.