How Do You Implement Rate Limiting Across Multiple Servers?
Learn how to implement rate limiting consistently across many servers using shared state, atomic operations, and sliding windows.
Expected Interview Answer
Distributed rate limiting means enforcing a shared request budget for a client across many stateless application servers, which requires moving the counter or token bucket out of any single server’s local memory into a shared, low-latency store like Redis so every instance sees the same up-to-date usage.
A per-server in-memory counter fails the moment traffic is load-balanced across multiple instances, because each server only sees a fraction of a client’s requests and the effective limit becomes (limit × number of servers). The fix is centralizing state in a fast shared store such as Redis, using an atomic operation like INCR with an expiry for fixed windows, or a Lua script that atomically checks and decrements a token bucket to avoid race conditions between concurrent requests. At very high scale, teams accept approximate limiting by sharding counters per server with periodic synchronization, or use a sliding window log/counter in Redis sorted sets for better accuracy than fixed windows without the cost of storing every timestamp. The key trade-offs are consistency versus latency (a synchronous check on every request adds a network hop) and exactness versus scalability (perfectly precise global limits get expensive at very high request rates, so many systems accept small overshoot for speed).
- Enforces a true global limit per client regardless of which server instance handles the request
- Atomic operations (INCR/Lua scripts) prevent race conditions from concurrent requests undercounting usage
- Centralized store enables consistent limits across horizontally scaled, stateless application servers
- Sliding window approaches in Redis avoid the burst problem of naive fixed windows
AI Mentor Explanation
Distributed rate limiting is like a bowler’s six-ball-per-over limit being tracked by one official scoreboard visible to every umpire on the field, instead of each umpire keeping their own private tally. If each umpire counted independently, the bowler could exploit the confusion and bowl far more than six deliveries across different umpires’ blind spots. By updating one shared, atomic counter that every umpire checks before allowing the next ball, the six-ball cap holds true no matter which umpire is watching at that moment. That single shared source of truth across many observers is exactly what a distributed rate limiter provides across many servers.
Step-by-Step Explanation
Step 1
Move state out of process memory
Replace per-server in-memory counters with a shared low-latency store like Redis so all instances see the same usage.
Step 2
Use atomic check-and-update operations
Use INCR with TTL, or a Lua script for token bucket logic, so concurrent requests cannot race past the limit.
Step 3
Pick a window strategy
Choose fixed window (simple, edge-burst prone), sliding window log/counter (more accurate), or token bucket (smooths bursts) based on accuracy needs.
Step 4
Decide on consistency trade-offs at scale
For very high throughput, accept approximate limiting via sharded local counters with periodic sync rather than a synchronous check on every request.
What Interviewer Expects
- Identifies that per-server in-memory counters break under load balancing across multiple instances
- Names a shared store (Redis) and an atomic mechanism (INCR+TTL or Lua script) to avoid race conditions
- Discusses window strategy trade-offs: fixed vs sliding vs token bucket in a distributed context
- Acknowledges the latency-vs-accuracy trade-off at very high scale and mentions approximate/sharded limiting
Common Mistakes
- Proposing per-server counters without addressing that load balancing splits traffic across them
- Ignoring race conditions from concurrent increments (not using atomic ops or Lua scripts)
- Not mentioning the added latency of a synchronous network call to the shared store on every request
- Forgetting the fixed-window burst problem when picking a window strategy at scale
Best Answer (HR Friendly)
“When you have many servers handling requests for the same client, you cannot let each server count independently, because the client could sneak past the limit by hitting different servers. Distributed rate limiting fixes this by keeping the actual count in one shared, fast store like Redis that every server checks, so the limit holds true no matter which server answers the request.”
Code Example
-- KEYS[1] = rate limit key, ARGV[1] = window in ms, ARGV[2] = limit, ARGV[3] = now
local key = KEYS[1]
local window = tonumber(ARGV[1])
local limit = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
redis.call("ZREMRANGEBYSCORE", key, 0, now - window)
local count = redis.call("ZCARD", key)
if count < limit then
redis.call("ZADD", key, now, now)
redis.call("PEXPIRE", key, window)
return 1 -- allowed
end
return 0 -- rejected, 429
-- Node.js caller:
// const allowed = await redis.eval(script, 1, `rate:${userId}`, windowMs, limit, Date.now())
// if (!allowed) return res.status(429).send("Too Many Requests")Follow-up Questions
- What happens to rate limiting accuracy if the shared Redis store itself becomes unavailable?
- How would you reduce the latency cost of a synchronous rate-limit check on every request?
- How does a sharded, approximate rate limiter trade accuracy for throughput at very high scale?
- How would you rate limit per-user AND per-IP simultaneously without doubling the network calls?
MCQ Practice
1. Why does a per-server in-memory rate limit counter fail once traffic is load-balanced across multiple instances?
Since each server tracks its own subset of requests, a client can exceed the intended global limit by being routed across servers.
2. What prevents a race condition when many concurrent requests check and update a shared rate-limit counter in Redis?
Atomic Redis operations or Lua scripts ensure the check-and-decrement happens as one indivisible step, avoiding lost updates.
3. What is the main trade-off of approximate, sharded rate limiting at very high scale?
Sharded local counters with periodic sync avoid a network hop on every request but allow some overshoot compared to a fully synchronous global check.
Flash Cards
Why do per-server counters fail for rate limiting? — Load balancing splits traffic, so each server only sees part of a client's requests, inflating the effective limit.
What store is commonly used for distributed rate limits? — A shared low-latency store like Redis, accessed via atomic operations.
How do you avoid race conditions in the shared counter? — Use atomic operations like INCR with TTL or a Lua script that checks and updates in one step.
Main trade-off at very high scale? — Exact synchronous global limits add latency; approximate sharded limiting trades some accuracy for speed.