What is a producer in Kafka and how does it choose a partition?
Learn what a Kafka producer is, how key hashing and the sticky partitioner select partitions, plus acks and idempotence, with a clear code example.
Expected Interview Answer
A Kafka producer is a client application that publishes records to topics; it chooses a partition by hashing the record key when one is present, or by using a sticky round-robin strategy when the key is null, and it can also target an explicit partition or use a custom partitioner.
The producer batches records, serializes them, and sends them to the leader of the target partition. When a record has a key, the default partitioner computes a hash so all records with the same key consistently land in the same partition, preserving their order. When the key is null, modern clients use a sticky partitioner that fills one batch before moving on, improving throughput. Delivery guarantees are controlled by the acks setting and idempotence.
- Key-based routing keeps related records ordered together
- Batching and compression boost throughput
- acks and retries tune the durability guarantee
- Idempotent producers avoid duplicate writes
- Custom partitioners allow application-specific routing
AI Mentor Explanation
A producer is like a scorer feeding deliveries into the right batter's log. When a delivery names a batter (the key), it always goes to that same batter's sheet so their innings stays in order. When no batter is named, the scorer just fills whichever sheet is currently open, spreading load evenly across the panel.
Step-by-Step Explanation
Step 1
Build and serialize the record
The producer creates a record with an optional key and value, then serializes both to bytes for the wire.
Step 2
Select a partition
If a key exists the default partitioner hashes it; if the key is null a sticky round-robin strategy picks the partition.
Step 3
Batch the record
Records for the same partition are grouped into a batch and optionally compressed to improve throughput.
Step 4
Send to the leader
The batch is sent to the leader broker of the chosen partition, which appends it and assigns offsets.
Step 5
Await acknowledgement
The acks setting decides whether the producer waits for the leader only or all in-sync replicas before confirming.
What Interviewer Expects
- Producer defined as a client that publishes records to topics
- Key hashing for partition selection and ordering
- Sticky round-robin behaviour when the key is null
- Awareness of custom and explicit partitioning
- Role of acks, retries and idempotence in delivery
Common Mistakes
- Saying a null key still guarantees ordering across partitions
- Confusing acks=all with idempotence
- Thinking the producer writes to followers directly
- Assuming every record is sent immediately with no batching
- Believing key routing depends on the consumer rather than the producer
Best Answer (HR Friendly)
“A Kafka producer is the part of an app that sends messages into Kafka. If a message has a key, the producer always sends messages with that key to the same slot so they stay in order; if there is no key, it spreads them out evenly to balance the load.”
Code Example
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
props.put("acks", "all");
props.put("enable.idempotence", "true");
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
// Keyed: all records for "user-42" hash to the same partition
producer.send(new ProducerRecord<>("orders", "user-42", "placed"));
// Null key: sticky round-robin across partitions
producer.send(new ProducerRecord<>("orders", null, "heartbeat"));
}Follow-up Questions
- What does the acks setting control in a producer?
- How does an idempotent producer prevent duplicate messages?
- When would you write a custom partitioner?
- What is the sticky partitioner and why was it introduced?
- How does batching affect producer throughput and latency?
MCQ Practice
1. How does the default partitioner route a record that has a key?
The default partitioner hashes the key so all records with the same key consistently reach the same partition.
2. What does the producer do when the record key is null?
With a null key, modern clients use a sticky partitioner that fills one batch before rotating to another partition.
3. Which setting controls how many replicas must acknowledge a write?
The acks setting determines whether the producer waits for the leader only or all in-sync replicas before confirming.
Flash Cards
What is a Kafka producer? — A client application that publishes records to Kafka topics and chooses their target partition.
How is a partition chosen for a keyed record? — The default partitioner hashes the key so identical keys always go to the same partition.
What happens with a null key? — The producer uses a sticky round-robin strategy to spread records across partitions.
What does acks control? — How many replicas must acknowledge a write before the producer treats it as successful.