How Do You Design Idempotent Writes in a Distributed System?
Learn how to design idempotent writes for distributed systems: conditional writes, dedupe tables, and at-least-once delivery.
Expected Interview Answer
Designing idempotent writes in a distributed system means ensuring that applying the same write operation multiple times — due to retries, at-least-once message delivery, or duplicate events — produces exactly the same end state as applying it once, typically by attaching a unique operation identifier or using naturally idempotent operations like conditional writes and versioned upserts instead of blind increments.
Distributed systems favor at-least-once delivery over exactly-once because true exactly-once delivery across a network is effectively impossible to guarantee cheaply, so duplicate messages, retried RPCs, and replayed events are a normal fact of life, not an edge case. The core technique is to make the write operation itself idempotent: instead of “increment balance by 10” (which produces a different result each time it is applied), use “set balance to 110 if current version is 4” (a conditional write keyed on a version or unique operation ID) or “insert this transaction record with unique ID X” where the database enforces uniqueness and silently no-ops on a duplicate insert. A dedupe table or exactly-once processing layer (storing processed message/operation IDs with a TTL) is a common building block on top of a message consumer, ensuring that even if the broker redelivers a message, the write handler recognizes and skips the duplicate. This matters most anywhere a consumer processes events from Kafka/SQS, a service retries a failed RPC to a downstream write API, or multiple replicas might independently attempt the same logical write during a failover.
- Makes the system safe under at-least-once delivery, the realistic default for distributed messaging
- Prevents silent data corruption from duplicate processing (double debits, double inventory decrements)
- Allows aggressive, simple retry logic everywhere without needing perfect delivery guarantees
- Enables safe failover and reprocessing after crashes without manual reconciliation
AI Mentor Explanation
Designing idempotent writes is like a scorer recording “the total after over 12 is 78 runs” instead of “add 6 runs” every time an update comes in — if the same over-12 update is announced twice due to a radio glitch, setting the absolute total to 78 again changes nothing, whereas blindly adding 6 twice would inflate the score. The scorer keeps a log of over numbers already recorded, so a repeated broadcast for over 12 is simply ignored the second time. This absolute, version-checked update instead of a relative increment is exactly how idempotent writes are designed in distributed systems.
Step-by-Step Explanation
Step 1
Assume at-least-once delivery
Treat every incoming write (message, retried RPC, replayed event) as potentially a duplicate of one already processed.
Step 2
Attach a unique operation ID
Every logical write carries a unique identifier (event ID, transaction ID, or version) that the store can check.
Step 3
Use conditional or absolute writes
Replace relative operations like increment/decrement with conditional writes (e.g., "set if version = N") or unique-constrained inserts.
Step 4
Maintain a dedupe record
Store processed operation IDs (often with a TTL) so a duplicate write is detected and skipped before touching business state.
What Interviewer Expects
- Explains why at-least-once delivery is the realistic default in distributed systems, not exactly-once
- Contrasts relative operations (increment) with idempotent conditional/absolute writes
- Describes a concrete dedupe mechanism: unique operation IDs, version checks, or unique DB constraints
- Applies this to real scenarios: Kafka/SQS consumers, retried RPCs, failover reprocessing
Common Mistakes
- Assuming the message broker or network can guarantee true exactly-once delivery
- Using blind increments/decrements for balances or counters that get replayed
- Forgetting to expire old dedupe records, causing an unbounded dedupe table
- Not handling the race where two duplicate writes arrive concurrently before the first is recorded
Best Answer (HR Friendly)
“In distributed systems, messages and requests sometimes get sent more than once by accident, so we design writes so that applying the same one twice has no extra effect. Instead of saying “add 10 to the balance,” we say “set the balance to this exact value if it hasn’t been set for this operation yet,” and we keep track of which operations we’ve already handled so duplicates are safely ignored.”
Code Example
def apply_transaction(db, transaction_id, account_id, new_balance, expected_version):
# 1. Check dedupe table first -- has this transaction already been applied?
if db.dedupe_table.exists(transaction_id):
return db.get_account(account_id) # safe replay, no-op
# 2. Conditional write: only succeeds if the version still matches
updated = db.accounts.update(
where={"id": account_id, "version": expected_version},
set={"balance": new_balance, "version": expected_version + 1},
)
if not updated:
raise ConflictError("version mismatch, retry with fresh read")
# 3. Record the transaction as processed, with a TTL for cleanup
db.dedupe_table.insert(transaction_id, ttl_seconds=86400)
return db.get_account(account_id)Follow-up Questions
- Why is exactly-once delivery generally considered impossible to guarantee cheaply across a network?
- How would you size and expire a dedupe table so it does not grow unbounded?
- How does optimistic concurrency control (version checks) relate to idempotent writes?
- How would you handle two duplicate writes for the same operation ID arriving concurrently?
MCQ Practice
1. Why must distributed write handlers be designed for idempotency?
At-least-once delivery is the realistic default in distributed messaging, so duplicate writes are a normal occurrence to design for.
2. Which write pattern is idempotent when replayed with the same input?
A conditional write keyed on version (or a unique operation ID) has no additional effect if replayed, unlike a blind increment.
3. What is a common building block for deduplicating distributed writes?
Storing processed operation IDs (with expiration) lets a handler detect and skip duplicate writes before touching business state.
Flash Cards
Why design for idempotent writes? — Because at-least-once delivery in distributed systems means writes can be duplicated by retries or redelivery.
Relative vs idempotent write? — "Increment by 10" is not idempotent; "set to 110 if version is 4" is idempotent.
What is a dedupe table? — A store of already-processed operation/message IDs (with TTL) used to detect and skip duplicate writes.
Where does this matter most? — Kafka/SQS consumers, retried RPCs to write APIs, and failover reprocessing after a crash.