How does Kafka handle message keys and what happens without a key?
Understand how Kafka message keys pick partitions, preserve per-key ordering, enable log compaction, and what happens when the key is null, with examples.
Expected Interview Answer
Kafka uses a message's key to decide its partition: it hashes the key and maps it to a partition, so all records sharing a key land on the same partition and keep their relative order. Without a key, records are distributed across partitions (sticky-batched round-robin in modern clients), giving load balance but no cross-record ordering guarantee.
The key is optional metadata separate from the value. When present, the default partitioner computes a hash of the key modulo the partition count, guaranteeing that, for example, all events for one user or order stay ordered on one partition. When the key is null, the producer spreads records for throughput and balance, so ordering only holds within whatever partition each record happens to reach. Keys also drive log compaction, where Kafka retains the latest value per key.
- Guarantees per-key ordering on one partition
- Co-locates related events for stateful processing
- Even load distribution when keys are absent
- Enables log compaction keyed by message key
- Predictable routing for the same entity over time
AI Mentor Explanation
The message key is like a batter's jersey number: every delivery tagged with that number is filed on the same page of the scorebook, so that batter's story stays in order. Leave the number blank and each ball is dropped onto whichever page has room — balanced across the book but with no guarantee one batter's deliveries stay together or in sequence.
Step-by-Step Explanation
Step 1
Producer sets the key
The application attaches an optional key (e.g. userId or orderId) alongside the message value.
Step 2
Hash the key
The default partitioner hashes the key (murmur2) and takes modulo the partition count to pick a partition.
Step 3
Same key, same partition
All records with the same key route to one partition, preserving their relative order there.
Step 4
Null key path
With no key the producer uses sticky partitioning to batch records, spreading load across partitions with no cross-record order.
Step 5
Downstream effects
Keys enable per-entity ordering, stateful consumer processing, and log compaction that keeps the latest value per key.
What Interviewer Expects
- Key drives partition selection via hashing
- Same key means same partition and preserved order
- Null key leads to balanced distribution, no ordering guarantee
- Awareness of sticky partitioning in modern clients
- Keys enable log compaction
Common Mistakes
- Believing Kafka guarantees global ordering rather than per-partition
- Thinking a null key means random single-record round-robin only
- Confusing the key with the message value
- Assuming key routing survives changing the partition count
- Forgetting keys are required for meaningful log compaction
Best Answer (HR Friendly)
“A Kafka message can carry a key that decides which partition it goes to, so all messages with the same key stay together and in order — handy for keeping one user's or order's events consistent. If you don't set a key, Kafka just spreads messages evenly for balance, and you lose that same-order guarantee across messages.”
Code Example
// Keyed: all events for order-42 go to the same partition, in order
producer.send(new ProducerRecord<>("orders", "order-42", eventJson));
// Null key: Kafka distributes across partitions (sticky-batched)
producer.send(new ProducerRecord<>("orders", null, eventJson));
// Default partition selection (conceptually):
// partition = hash(key) % numPartitions when key != nullFollow-up Questions
- Why does adding partitions break existing key-to-partition mapping?
- How do message keys relate to log compaction?
- What is sticky partitioning and why was it introduced?
- How would you guarantee ordering for all events of one customer?
- Can two different keys ever share a partition?
MCQ Practice
1. What determines a keyed message's partition by default?
The default partitioner hashes the key and takes modulo the number of partitions, so equal keys map to the same partition.
2. What ordering guarantee do you get when messages have no key?
Without a key, records spread across partitions, so ordering only holds within each partition, not across related records.
3. Which Kafka feature relies on message keys to keep the latest value per key?
Log compaction retains the most recent value for each key, so keys are essential for it to work meaningfully.
Flash Cards
What decides a keyed message's partition? — hash(key) % number_of_partitions via the default partitioner.
Same key guarantee? — All records with the same key go to one partition and keep their relative order.
What happens with a null key? — Records are spread across partitions (sticky-batched) for balance, with no cross-record ordering.
Why do keys matter for compaction? — Log compaction keeps only the latest value per key, so a key is required for it to be useful.