How to Design a Rate Limiter?
Learn how to design a rate limiter for system design interviews: token bucket, leaky bucket, sliding window, and distributed Redis counters.
Expected Interview Answer
A rate limiter is designed by choosing an algorithm (token bucket, leaky bucket, fixed window, or sliding window log) that tracks request counts per client key over time and rejects or delays requests once a threshold is exceeded, backed by a fast shared store like Redis for distributed enforcement.
The core decision is the algorithm: token bucket allows controlled bursts by refilling tokens at a steady rate, leaky bucket smooths traffic into a constant outflow, fixed window is simple but allows edge bursts at window boundaries, and sliding window log or counter gives the most accurate limiting at higher storage cost. In a distributed system the counters must live in a shared, low-latency store such as Redis, using atomic operations (INCR with expiry, or a Lua script) so multiple API gateway instances agree on the same count. The limiter typically keys on a combination of user ID, API key, or IP address, and returns HTTP 429 with a Retry-After header when the limit is hit. Production systems also add multiple tiers (per-second burst plus per-day quota) and graceful degradation if the counting store itself becomes unavailable.
- Protects backend services from being overwhelmed by traffic spikes
- Enforces fair usage and API quotas across tenants
- Prevents abuse patterns like credential stuffing or scraping
- Keeps latency predictable by shedding excess load early
AI Mentor Explanation
A rate limiter is like the twenty-over cap in a T20 innings: a batting side can only face a fixed number of deliveries no matter how well set they are, and once the quota of overs is used up, the innings simply stops regardless of remaining intent. The umpire tracks the ball count every over exactly like a shared counter tracks requests per window. If a team tries to bowl a no-ball or extra delivery beyond the allotted overs, it is called back — the same way a rate limiter rejects a request past the threshold. This hard ceiling on deliveries per innings is precisely the fixed-window idea behind API throttling.
Step-by-Step Explanation
Step 1
Choose an algorithm
Pick token bucket, leaky bucket, fixed window, or sliding window based on burst tolerance and accuracy needs.
Step 2
Pick the limiting key
Key counters by user ID, API key, or IP so limits apply per client, not globally.
Step 3
Use a shared, atomic store
Back counters with Redis (INCR + expiry or a Lua script) so all gateway instances agree.
Step 4
Respond and degrade gracefully
Return 429 with Retry-After on breach, and fail open or use local fallback if the store is down.
What Interviewer Expects
- Naming at least two limiting algorithms and their trade-offs
- Discussing distributed counting via a shared store like Redis
- Choosing a sensible limiting key (user, API key, or IP)
- Addressing failure modes: store outage, clock skew, thundering herd at window reset
Common Mistakes
- Assuming a single in-memory counter works across multiple servers
- Ignoring burst behavior at fixed-window boundaries
- Not returning a Retry-After header or clear error semantics
- Forgetting what happens when the rate-limiting store itself fails
Best Answer (HR Friendly)
“A rate limiter protects a service by capping how many requests a client can make in a given time window. I would pick an algorithm like token bucket for burst tolerance, track counts in a shared store like Redis so it works across multiple servers, and return a clear 429 error with a retry hint once a client goes over their limit.”
Code Example
async function allowRequest(redis, key, capacity, refillPerSec) {
const now = Date.now() / 1000;
const bucket = await redis.hgetall(key);
let tokens = bucket.tokens ? parseFloat(bucket.tokens) : capacity;
const lastRefill = bucket.ts ? parseFloat(bucket.ts) : now;
const elapsed = now - lastRefill;
tokens = Math.min(capacity, tokens + elapsed * refillPerSec);
if (tokens < 1) {
await redis.hset(key, { tokens, ts: now });
return { allowed: false, retryAfterSec: (1 - tokens) / refillPerSec };
}
tokens -= 1;
await redis.hset(key, { tokens, ts: now });
await redis.expire(key, 60);
return { allowed: true };
}Follow-up Questions
- How would you rate-limit across multiple data centers with minimal latency?
- What is the difference between fixed window and sliding window log limiting?
- How do you avoid the thundering herd problem at a fixed-window reset boundary?
- How would you rate-limit by user tier (free vs. paid) differently?
MCQ Practice
1. Which rate-limiting algorithm allows controlled bursts by letting requests consume pre-accumulated tokens?
Token bucket lets a client burst up to the number of tokens currently available, then throttles once tokens run out.
2. Why is a shared store like Redis typically used for rate limiting in a distributed system?
Without a shared counter, each server would track limits independently and the true per-client limit could be exceeded.
3. What HTTP status code should a rate limiter typically return when a client exceeds its quota?
429 Too Many Requests signals the client has exceeded its rate limit, ideally with a Retry-After header.
Flash Cards
What does a rate limiter do? — Caps the number of requests a client can make in a time window to protect backend capacity.
Token bucket in one line? — Tokens refill steadily; each request consumes one, allowing bursts up to the bucket size.
Why use Redis for rate limiting? — It gives a fast, shared, atomic counter across multiple stateless gateway instances.
Fixed window drawback? — It can allow up to 2x the intended rate right at window boundaries.