How Would You Design a Distributed Lock?
Learn how to design a distributed lock: atomic acquisition, TTL expiry, fencing tokens, and Redis vs ZooKeeper trade-offs.
Expected Interview Answer
A distributed lock is a coordination primitive that lets multiple independent processes across different machines agree that only one of them may hold a named resource at a time, typically implemented with a shared store like Redis or a consensus system like ZooKeeper or etcd, using atomic set-if-not-exists with an expiry plus a fencing token to guard against stale lock holders.
A single node acquires the lock by atomically writing a unique token to a key only if it does not already exist, with a time-to-live so a crashed holder does not block others forever. Because network delays or GC pauses can make a process believe it still holds the lock after its lease expired, a robust design attaches a monotonically increasing fencing token to every lock grant, and downstream resources reject any write that carries an older token than one already seen. Simple single-node Redis locks (like naive SETNX) are vulnerable to failover races, so production-grade designs either use a consensus-based store (ZooKeeper ephemeral sequential nodes, etcd leases) or Redlock-style majority acquisition across independent Redis nodes. The key design tension is safety (never let two holders proceed concurrently) versus liveness (never deadlock if a holder crashes) versus performance (avoid a slow consensus round trip for every lock).
- Prevents two processes from concurrently mutating a shared resource across machines
- TTL/lease expiry guarantees liveness even if the lock holder crashes
- Fencing tokens protect against stale processes acting after their lease has actually expired
- Consensus-backed implementations give strong safety guarantees for critical operations
AI Mentor Explanation
A distributed lock is like the single physical bat that only one batter can hold at a time, no matter how many players want to score. A batter claims the bat by writing their name in the official scorebook (atomic set-if-not-exists), and umpires assign each claim a rising innings number (fencing token) so a batter who returns to the crease after being confused about the over cannot score using an old, invalid claim. If a batter walks off injured without returning the bat, a shot clock (TTL) forces the umpire to reissue it after a timeout so the game does not stall forever. That single-holder claim with a timeout and an increasing sequence number is exactly how a distributed lock works.
Step-by-Step Explanation
Step 1
Acquire atomically with a TTL
A process attempts an atomic SET-if-not-exists on a shared key with a unique value and an expiry, so a crash cannot deadlock other holders.
Step 2
Attach a fencing token
The lock service issues a monotonically increasing token with each successful grant, used to reject stale writes.
Step 3
Do the protected work
The holder performs its critical-section operations, presenting the fencing token to any downstream resource it writes to.
Step 4
Release or expire
The holder releases the lock explicitly (compare-and-delete by its unique value) or the TTL expires, freeing it for the next claimant.
What Interviewer Expects
- Explains atomic acquire (SETNX-style) with an expiry/TTL, not just “use a lock”
- Raises the stale-lock-holder problem and proposes fencing tokens as the fix
- Names a real system: Redis (Redlock or single-instance with caveats), ZooKeeper, or etcd
- Discusses the safety vs liveness trade-off and why naive locks can fail under GC pauses/network delays
Common Mistakes
- Assuming a simple SETNX lock is safe without discussing TTL expiry or fencing tokens
- Not addressing what happens if the lock holder crashes mid-critical-section
- Ignoring clock drift / GC pause issues that make TTL-based locks unsafe without fencing
- Confusing a distributed lock with a local mutex, missing the network partition dimension entirely
Best Answer (HR Friendly)
“A distributed lock lets separate machines agree that only one of them is allowed to touch a shared resource at a time, even though they cannot see each other directly. I would use a shared coordination store like Redis or ZooKeeper where a process atomically claims a key with an expiration, and pair that with an increasing token number so that even if a process is delayed and thinks it still holds the lock, downstream systems can detect and reject its stale action.”
Code Example
import time
import uuid
def acquire_lock(redis, resource, ttl_ms=10000):
token = str(uuid.uuid4())
acquired = redis.set(
name=f"lock:{resource}",
value=token,
nx=True,
px=ttl_ms,
)
if not acquired:
return None
fencing_token = redis.incr(f"fence:{resource}")
return {"token": token, "fence": fencing_token}
def release_lock(redis, resource, token):
# Compare-and-delete via Lua to avoid releasing someone else's lock
script = """
if redis.call("get", KEYS[1]) == ARGV[1] then
return redis.call("del", KEYS[1])
end
return 0
"""
return redis.eval(script, 1, f"lock:{resource}", token)
def write_with_fence(storage, resource, fencing_token, data):
# Downstream store rejects writes from an older fencing token
if fencing_token < storage.get_last_fence(resource):
raise Exception("stale lock holder: rejected")
storage.write(resource, data, fencing_token)Follow-up Questions
- What is the Redlock algorithm and why is it controversial for strong safety guarantees?
- How does ZooKeeper’s ephemeral sequential node approach differ from a Redis TTL lock?
- How would you prevent a lock holder that is paused by GC from corrupting data after its lease expires?
- When would you choose optimistic concurrency control over a distributed lock entirely?
MCQ Practice
1. What problem do fencing tokens solve in distributed locking?
Fencing tokens let downstream resources reject stale writes from a process whose lease actually expired but who has not realized it yet.
2. Why does a distributed lock need a TTL/expiry on acquisition?
Without an expiry, a crashed process holding the lock would deadlock every other process waiting on that resource.
3. Which of these is a coordination system commonly used to implement robust distributed locks?
ZooKeeper (and similarly etcd) provides consensus-backed primitives such as ephemeral sequential nodes suited for distributed locking.
Flash Cards
What is a distributed lock? — A coordination primitive ensuring only one process across machines holds a resource at a time.
Why use a TTL on lock acquisition? — So a crashed holder does not block the resource forever; the lock auto-expires.
What is a fencing token? — A monotonically increasing number issued per lock grant, used to reject writes from stale holders.
Name two systems used to implement distributed locks. — Redis (with TTL/Redlock) and ZooKeeper/etcd (consensus-based).