Idempotency in Distributed Systems
In distributed systems, networks fail in ambiguous ways: a client sends a request, the server processes it, but the response is lost before the client receives it. The client cannot tell whether the operation succeeded and simply needs its result re-delivered, or whether it never arrived at all. The only safe general strategy is to retry — but retries are only safe if the operation is idempotent, meaning that executing it two or more times has the exact same effect as executing it once. GET requests are naturally idempotent because they do not change state. The hard cases are operations that mutate state, like charging a credit card or decrementing inventory, where a naive retry can cause the operation to happen twice.
Cricket analogy: When a DRS review signal from the third umpire gets lost in transmission, the on-field umpire re-requests it, safe only because replaying the same boundary review doesn't change the runs already awarded, unlike a repeated run-addition would.
Idempotency Keys
The standard mechanism for making a non-idempotent operation safe to retry is an idempotency key: the client generates a unique identifier (often a UUID) for a logical operation before it is first attempted, and includes that key on every retry of that same operation. The server stores a record of keys it has already processed, along with the result. When a request arrives with a key already on record, the server returns the stored result without re-executing the underlying side effect. This shifts the deduplication responsibility to the server and makes the client's job simple: keep resending the same request with the same key until an acknowledgment is received.
Cricket analogy: Each DRS review is tagged with a unique review number for that delivery, so if the same review number arrives twice, the third umpire's team just replays the stored verdict instead of re-analyzing the ball.
Natural vs. Engineered Idempotency
Some operations are naturally idempotent by construction: setting a field to an absolute value ('set balance to $50') can be repeated safely, because the end state is the same regardless of how many times it runs. Operations expressed as deltas ('add $50 to balance') are not naturally idempotent, since repeating them changes the result each time. Where possible, designing operations to be naturally idempotent (absolute sets, conditional writes, upserts keyed by a stable identifier) avoids the need for extra bookkeeping. Where the operation is inherently a delta — like a payment charge — engineered idempotency via keys or deduplication tables is required.
Cricket analogy: Setting a scoreboard to 'target: 250' can be redisplayed any number of times with the same result, but an 'add 4 runs' instruction run twice by mistake would wrongly credit 8 runs for one boundary.
def charge_customer(idempotency_key, customer_id, amount_cents):
existing = db.get_idempotency_record(idempotency_key)
if existing is not None:
# Already processed this logical request; return the stored
# result instead of charging again.
return existing.result
# Reserve the key atomically before doing the side effect, so a
# concurrent retry sees the reservation and waits/returns early
# instead of racing to charge twice.
if not db.try_reserve_idempotency_key(idempotency_key):
raise ConcurrentRetryInProgress()
result = payment_gateway.charge(customer_id, amount_cents)
db.save_idempotency_record(idempotency_key, result)
return resultStripe's API popularized idempotency keys for payments: a client sends an Idempotency-Key header on a POST /charges request, and Stripe guarantees that retrying the exact same request with the same key within a 24-hour window will not create a second charge, returning the original charge's result instead. This lets client libraries retry aggressively on network timeouts without any risk of double-billing.
A frequent bug is reserving the idempotency key and performing the side effect in two separate, non-atomic steps without protecting the window between them — a concurrent retry arriving during that window can slip through and cause a duplicate charge. The reservation and the intent to execute must be committed together, typically via a unique constraint or conditional write in the datastore, not just an application-level check.
Idempotency vs. Exactly-Once Delivery
It is a common misconception that idempotency and exactly-once delivery are the same thing. Message brokers overwhelmingly provide at-least-once delivery: a message may be redelivered after a consumer crash before it acknowledges processing. True exactly-once delivery across a network is extremely difficult to guarantee in general. Idempotency is the practical way systems achieve exactly-once effect on top of at-least-once delivery — the message may arrive multiple times, but the consumer's idempotent handling ensures it only takes effect once.
Cricket analogy: Ball-tracking data might be re-transmitted to the broadcast graphics team after a dropout, so the graphics operator relies on the system recognizing the duplicate and rendering the trajectory only once.
- Idempotency means repeating an operation has the same effect as performing it once, making retries safe.
- Idempotency keys let a server recognize and deduplicate retried requests for the same logical operation.
- Naturally idempotent operations (absolute sets, upserts) need no extra machinery; delta-based operations (charges, increments) require engineered idempotency.
- Reserving the idempotency key and executing the side effect must be atomic to prevent a race during concurrent retries.
- At-least-once delivery plus idempotent handling is how most real systems achieve an effective exactly-once outcome.
- Stripe's Idempotency-Key header is a widely cited real-world implementation pattern for safe payment retries.
Practice what you learned
1. What does it mean for an operation to be idempotent?
2. Why is 'add $50 to balance' not naturally idempotent while 'set balance to $50' is?
3. What is the purpose of an idempotency key in a payment API like Stripe's?
4. Why must reserving an idempotency key and performing the associated side effect be atomic?
5. How do most real-world distributed systems achieve an effectively exactly-once outcome?
Was this page helpful?
You May Also Like
Event-Driven Architecture
A design style where services communicate by producing and reacting to events rather than calling each other directly, enabling loose coupling and independent scaling.
Message Queues Explained
Message queues decouple producers from consumers by buffering work as discrete messages, enabling asynchronous processing, load leveling, and resilience to downstream slowdowns.
Distributed Transactions
Distributed transactions coordinate atomic, all-or-nothing updates across multiple independent services or databases, using protocols like two-phase commit or pattern-based sagas.
Circuit Breakers and Bulkheads
Resilience patterns that stop cascading failures — circuit breakers halt calls to a failing dependency, while bulkheads isolate resource pools so one failure cannot exhaust everything.
Related Reading
Related Study Notes in Software Engineering
Browse all study notesMicroservices Study Notes
Software Architecture · 30 topics
Software EngineeringTesting & TDD Study Notes
Software Testing · 30 topics
Software EngineeringDesign Patterns Study Notes
Software Design · 30 topics
Software EngineeringSoftware Engineering Study Notes
Python · 40 topics
Software EngineeringGit & Version Control Study Notes
Bash · 40 topics