What is idempotent producer in Kafka and how does it prevent duplicates?
Understand the Kafka idempotent producer, how producer IDs and sequence numbers prevent duplicate writes, its config, limits and interview questions.
Expected Interview Answer
An idempotent producer in Kafka guarantees that retried message sends are written to a partition exactly once, preventing the duplicates that ordinary retries would create. It is enabled with enable.idempotence=true and works by tagging every batch with a producer ID and a monotonically increasing sequence number the broker uses to detect and discard repeats.
When idempotence is on, the broker assigns each producer a unique producer ID (PID) and the producer stamps each record batch per partition with an incrementing sequence number. The broker keeps the last sequence number it accepted for that PID and partition, so if a retry arrives with an already-seen sequence number it is acknowledged but not written again. This makes retries safe and preserves ordering. It requires acks=all, and Kafka automatically enforces max.in.flight.requests.per.connection <= 5 so ordering and dedup hold. Idempotence covers duplicates within a single producer session to one partition; it is not the same as cross-partition, multi-message transactions.
- Exactly-once write semantics per partition despite retries
- Eliminates duplicates caused by network timeouts and re-sends
- Preserves message ordering within a partition
- Enabled with a single config flag, no application code changes
- Foundation that Kafka transactions build upon
AI Mentor Explanation
Imagine a scorer who numbers every run entry sequentially. If a shaky phone line makes them re-send 'run number 47' twice, the official scorebook sees 47 already recorded and ignores the copy. Kafka's idempotent producer works this way: each batch carries a producer ID and a rising sequence number, so a retried send with a number the broker has already logged is acknowledged but never double-counted on the scoreboard.
Step-by-Step Explanation
Step 1
Enable idempotence
Set enable.idempotence=true; Kafka then enforces acks=all and bounded in-flight requests.
Step 2
Broker assigns a PID
On init the producer gets a unique producer ID that identifies its session to the broker.
Step 3
Stamp sequence numbers
Each per-partition batch gets a monotonically increasing sequence number tied to the PID.
Step 4
Broker deduplicates
The broker stores the last accepted sequence per PID and partition, discarding any repeat it has already written.
Step 5
Safe retries
Timed-out sends can be retried without creating duplicates, and ordering is preserved.
What Interviewer Expects
- That enable.idempotence=true is the switch
- The role of producer ID plus per-partition sequence numbers
- Why acks=all and bounded in-flight requests are required
- That it is exactly-once per partition within one producer session
- The distinction between idempotence and full transactions
Common Mistakes
- Thinking idempotence gives cross-partition exactly-once (that needs transactions)
- Believing it deduplicates across different producer sessions or restarts
- Forgetting it requires acks=all
- Confusing broker-side dedup with application-level deduplication
- Assuming it prevents duplicates a consumer might create on reprocessing
Best Answer (HR Friendly)
“An idempotent producer makes sure that if Kafka has to resend a message after a network glitch, it still gets stored only once instead of twice. It does this by giving each message a unique number so the server can spot and ignore repeats, and you turn it on with a single setting.”
Code Example
Properties props = new Properties();
props.put("bootstrap.servers", "localhost:9092");
props.put("key.serializer", StringSerializer.class.getName());
props.put("value.serializer", StringSerializer.class.getName());
// Turning this on auto-sets acks=all and caps in-flight requests
props.put("enable.idempotence", true);
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
// Retried sends now carry the same sequence number and are deduped by the broker
producer.send(new ProducerRecord<>("payments", "txn-42", "amount=100"));
producer.flush();
producer.close();Follow-up Questions
- How does the broker use the producer ID and sequence number to detect a duplicate?
- Why does idempotence require max.in.flight.requests.per.connection to be five or fewer?
- What happens to the producer ID when the producer restarts?
- How do idempotent producers relate to Kafka transactions and exactly-once processing?
- Does idempotence protect against duplicates introduced on the consumer side?
MCQ Practice
1. Which config enables an idempotent producer?
Setting enable.idempotence=true turns on producer IDs and sequence numbers and enforces acks=all.
2. How does the broker detect a duplicate from an idempotent producer?
The broker tracks the last accepted sequence number per producer ID and partition, discarding repeats.
3. What is the scope of idempotence guarantees?
Idempotence prevents duplicate writes per partition for a single producer session; cross-partition atomicity needs transactions.
Flash Cards
Enable idempotence? — Set enable.idempotence=true (auto-sets acks=all).
How are duplicates detected? — Producer ID (PID) plus a per-partition monotonic sequence number.
Ordering constraint? — Kafka caps max.in.flight.requests.per.connection at 5 to keep order and dedup.
Scope of the guarantee? — Exactly-once per partition within one producer session.
Idempotence vs transactions? — Idempotence dedups writes per partition; transactions add atomic multi-partition commits.
Continue Learning
Related Interview Questions
What are the different delivery semantics in Kafka (at-most-once, at-least-once, exactly-once)?
hard
What is acks configuration in a Kafka producer and what do 0, 1, and all mean?
medium
What is a transactional producer in Kafka?
hard
Where do Kafka's exactly-once semantics stop, and what must the application still handle?
hard