What are the different delivery semantics in Kafka (at-most-once, at-least-once, exactly-once)?
Understand Kafka's at-most-once, at-least-once, and exactly-once delivery semantics, the producer and consumer configs behind each, and their trade-offs.
Expected Interview Answer
Kafka supports three delivery guarantees: at-most-once (messages may be lost but never duplicated), at-least-once (messages are never lost but may be duplicated), and exactly-once (each message is processed once with no loss or duplication).
The guarantee depends on how producers and consumers are configured. At-most-once comes from committing offsets before processing or using acks=0. At-least-once — the common default — comes from acks=all with retries plus committing offsets only after processing, so a crash triggers reprocessing. Exactly-once semantics (EOS) is achieved with idempotent producers, transactions that atomically write records and commit offsets, and consumers reading with isolation.level=read_committed, giving a read-process-write pipeline that neither loses nor duplicates.
- Match the guarantee to the cost of loss vs duplication
- At-least-once gives durability with simple config
- Exactly-once removes the need for downstream deduplication
- Transactions make multi-topic writes atomic
- read_committed hides aborted transactional records
AI Mentor Explanation
At-most-once is a scorer who writes the run before checking it happened — fast, but a miss means a lost run with no double counting. At-least-once is a scorer who confirms every run and re-asks if unsure, so nothing is missed but a run might get written twice. Exactly-once is a scorer using a signed, numbered slip per delivery: the umpire and scorer commit together atomically, so each run is recorded once — never lost, never duplicated, even if someone drops the pen mid-over.
Step-by-Step Explanation
Step 1
Pick the guarantee you need
Decide whether loss or duplication is more acceptable for your use case.
Step 2
Configure at-most-once
Commit offsets before processing (or use acks=0) so a crash skips rather than repeats.
Step 3
Configure at-least-once
Use acks=all with retries and commit offsets only after successful processing.
Step 4
Enable idempotent producer
Set enable.idempotence=true to remove producer-side duplicates within a partition.
Step 5
Use transactions for exactly-once
Wrap sends and sendOffsetsToTransaction in a transaction and read with isolation.level=read_committed.
What Interviewer Expects
- Clear definition of all three semantics
- Producer configs behind each (acks, retries, idempotence)
- Consumer offset-commit timing
- How transactions and read_committed enable EOS
- Trade-offs in throughput and complexity
Common Mistakes
- Confusing at-least-once with exactly-once
- Thinking idempotence alone gives end-to-end exactly-once
- Committing offsets before processing yet expecting no loss
- Forgetting consumers must use read_committed for EOS
Best Answer (HR Friendly)
“Kafka can deliver messages in three ways: at-most-once means you might lose a few but never repeat any, at-least-once means you never lose any but might see repeats, and exactly-once means each message is handled precisely one time. You choose by configuring how producers acknowledge writes and when consumers save their progress.”
Code Example
Properties props = new Properties();
props.put("enable.idempotence", "true");
props.put("acks", "all");
props.put("transactional.id", "orders-eos-1");
KafkaProducer<String, String> producer = new KafkaProducer<>(props);
producer.initTransactions();
try {
producer.beginTransaction();
producer.send(new ProducerRecord<>("orders-out", "order-42", "PROCESSED"));
// atomically commit the consumed offsets with the produced records
producer.sendOffsetsToTransaction(currentOffsets, consumerGroupMetadata);
producer.commitTransaction();
} catch (KafkaException e) {
producer.abortTransaction();
}Follow-up Questions
- How does sendOffsetsToTransaction make a read-process-write atomic?
- What does isolation.level=read_committed do on the consumer?
- Why is idempotence necessary but not sufficient for exactly-once?
- When would at-most-once actually be preferable?
- What is the performance cost of enabling transactions?
MCQ Practice
1. Which semantic may lose messages but never duplicates them?
At-most-once commits before processing or skips retries, so a failure loses the message without duplicating it.
2. What combination enables exactly-once processing in Kafka?
EOS needs idempotent producers, transactional writes that also commit offsets, and consumers reading only committed records.
3. The common default guarantee with acks=all, retries, and commit-after-process is?
That configuration never loses messages but can reprocess after a crash, which is at-least-once.
Flash Cards
At-most-once — Messages may be lost but are never duplicated — commit offsets before processing or use acks=0.
At-least-once — Messages are never lost but may be duplicated — acks=all, retries, commit offsets after processing.
Exactly-once — Each message processed once — idempotent producer + transactions + read_committed consumer.
Why read_committed? — Consumers skip records from aborted transactions, so only successfully committed data is seen.
Continue Learning
Related Interview Questions
What is idempotent producer in Kafka and how does it prevent duplicates?
medium
Where do Kafka's exactly-once semantics stop, and what must the application still handle?
hard
What is acks configuration in a Kafka producer and what do 0, 1, and all mean?
medium
What is a transactional producer in Kafka?
hard