Rate Limiting Strategies
Rate limiting restricts how many requests a client, user, API key, or IP address may make within a given time period, and it serves several purposes simultaneously: it protects backend capacity from being overwhelmed by a traffic spike or bug in a client, it prevents abuse such as credential-stuffing or scraping, and it enforces fair usage across tenants in a multi-tenant system so that one noisy client cannot starve others. Rate limiting is typically enforced at the edge — in an API gateway or load balancer — so that rejected requests never consume backend resources, but it can also be applied at any layer where a shared resource needs protecting.
Cricket analogy: A stadium's turnstile system caps how many fans per gate can enter per minute, protecting the concourse from being overwhelmed and ensuring one gate's rush doesn't starve others of staff attention, enforced right at the gate rather than deep inside the stadium.
Fixed Window and Its Boundary Problem
The simplest algorithm, fixed window counting, divides time into fixed intervals (e.g. one-minute windows) and allows up to N requests per window, resetting the counter at each boundary. It is cheap to implement and reason about, but it has a well-known boundary flaw: a client can send N requests in the last instant of one window and another N requests in the first instant of the next window, resulting in nearly 2N requests within a much shorter span than the intended window — effectively doubling the allowed burst rate right at window edges.
Cricket analogy: A stadium allowing exactly 500 entries per fixed 10-minute window can end up admitting nearly 1000 fans in a short span if a crowd rushes the gate at the last second of one window and again at the first second of the next.
Sliding Window, Token Bucket, and Leaky Bucket
Sliding window log and sliding window counter algorithms fix the boundary problem by considering a continuously moving window rather than discrete resets — the log approach tracks exact timestamps of recent requests (accurate but memory-heavy), while the counter approach approximates the sliding window by weighting the previous and current fixed windows proportionally to elapsed time (cheap and accurate enough for most uses). Token bucket allows bursts up to a bucket capacity while enforcing a steady average refill rate: each request consumes a token, tokens refill continuously up to the bucket's capacity, and requests are rejected once the bucket is empty — this is popular because it naturally accommodates bursty-but-average-bounded traffic. Leaky bucket instead enforces a strictly constant outflow rate by queuing incoming requests and processing them at a fixed rate, smoothing bursts entirely rather than allowing them.
Cricket analogy: A bowler's spell is like a leaky bucket enforcing a strict constant over rate, while a batting side accumulating boundary 'tokens' that let them burst with three sixes in an over but average out over the innings resembles a token bucket.
class TokenBucket:
def __init__(self, capacity, refill_rate_per_sec):
self.capacity = capacity
self.tokens = capacity
self.refill_rate = refill_rate_per_sec
self.last_refill = time.monotonic()
def allow_request(self):
now = time.monotonic()
elapsed = now - self.last_refill
# Refill tokens proportional to elapsed time, capped at capacity
self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
self.last_refill = now
if self.tokens >= 1:
self.tokens -= 1
return True # request allowed
return False # request rejected: bucket emptyStripe's public API documents a token-bucket-style rate limiter per API key, and returns a 429 status with a Retry-After header telling well-behaved clients exactly how long to back off. This combination — reject with a clear signal rather than silently dropping or queueing indefinitely — is a widely adopted convention that lets client libraries implement automatic, respectful retries.
A common design mistake is rate limiting only by IP address in a system where many legitimate users sit behind a shared NAT or corporate proxy — all of them get throttled together as if they were one client. Rate limits should generally key on the most specific identity available (API key, authenticated user ID, or a composite key) and fall back to IP only when no better identity exists, such as for anonymous traffic.
Distributed Rate Limiting
In a system with many stateless gateway instances, an in-memory counter on each instance undercounts the true global rate, since each instance only sees a fraction of traffic. Distributed rate limiting solves this by centralizing counters in a shared, low-latency store such as Redis, using atomic increment-and-expire operations so concurrent requests across instances are counted consistently. This adds a network round trip per request and a shared dependency, so systems at very high scale sometimes accept approximate limiting (e.g. local counters combined with periodic synchronization) to trade strict accuracy for lower latency and reduced load on the shared store.
Cricket analogy: If each stand's local ticket scanner only counted its own gate's entries, the stadium would undercount the true crowd; centralizing the count in one shared ledger that every gate atomically updates enforces the true capacity limit across all gates.
- Rate limiting protects capacity, deters abuse, and enforces fair usage across clients or tenants.
- Fixed window counting is simple but allows nearly double the intended burst rate at window boundaries.
- Sliding window log/counter algorithms fix the boundary problem at the cost of extra memory or approximation.
- Token bucket permits bursts up to a capacity with a steady average refill rate; leaky bucket smooths traffic to a strictly constant output rate.
- Rate limits should key on the most specific client identity available (API key/user ID) rather than IP alone, to avoid punishing shared-NAT users.
- Distributed rate limiting requires a shared, atomic counter store (e.g. Redis) so limits are enforced accurately across many stateless gateway instances.
Practice what you learned
1. What is the main flaw of the fixed window rate limiting algorithm?
2. How does token bucket rate limiting differ from leaky bucket?
3. Why is rate limiting purely by IP address problematic?
4. Why does distributed rate limiting typically require a shared store like Redis?
5. What does a 429 response with a Retry-After header communicate to a client?
Was this page helpful?
You May Also Like
Designing a Rate Limiter
Walks through building a distributed rate limiter that protects APIs from abuse, comparing token bucket, sliding window, and fixed window algorithms and where state should live.
Circuit Breakers and Bulkheads
Resilience patterns that stop cascading failures — circuit breakers halt calls to a failing dependency, while bulkheads isolate resource pools so one failure cannot exhaust everything.
Cache Eviction Policies
Covers the algorithms caches use to decide what to remove when full, comparing LRU, LFU, FIFO, and TTL-based approaches and their tradeoffs.
Consistent Hashing
Introduces the hashing technique that minimizes data movement when nodes are added or removed, forming the backbone of distributed caches, sharded databases, and load balancers.
Related Reading
Related Study Notes in Software Engineering
Browse all study notesMicroservices Study Notes
Software Architecture · 30 topics
Software EngineeringTesting & TDD Study Notes
Software Testing · 30 topics
Software EngineeringDesign Patterns Study Notes
Software Design · 30 topics
Software EngineeringSoftware Engineering Study Notes
Python · 75 topics
Software EngineeringGit & Version Control Study Notes
Bash · 40 topics