Rate Limiting in Microservices

Analogy categoryAnalogies

Why Rate Limit Between Services?

Rate limiting isn't only about stopping external abusive clients; inside a microservices architecture it's just as important for protecting services from each other. A batch job that suddenly starts calling an internal API in a tight loop, a misconfigured retry policy, or a legitimate traffic spike from one high-volume tenant can all overwhelm a downstream service just as effectively as a malicious actor. Rate limiting enforces an agreed-upon ceiling on request volume per client, per API key, or per tenant, so that capacity planning becomes predictable and one caller's burst can't degrade service for everyone else sharing that dependency.

🏏

Cricket analogy: A stadium caps ticket sales per match at the ground's actual seating capacity rather than selling unlimited tickets, because unchecked sales would create a crush at the gates, just as unrate-limited calls can crush a shared service.

Rate Limiting Algorithms

The token bucket algorithm maintains a bucket that refills with tokens at a fixed rate (say, 100 per second) up to a maximum capacity; each request consumes one token, and requests are rejected once the bucket is empty, which naturally allows short bursts up to the bucket size while enforcing a steady average rate over time. The sliding window log and sliding window counter algorithms instead track request counts over a moving time window (e.g., the last 60 seconds) to avoid the boundary problem of fixed windows, where a client could send double the intended limit by timing requests around a window edge — for example, sending the full quota in the last second of one minute and the full quota again in the first second of the next.

🏏

Cricket analogy: The token bucket is like a bowler's over allowance: they can bowl six balls (tokens) in a burst within the over, but once used, they must wait for the next over (refill) rather than bowling unlimited deliveries back to back.

Where to Enforce Rate Limits

Rate limits can be enforced at the API gateway (protecting the whole backend from any single client, typically keyed by API key or IP), at the individual service level (protecting one service's specific capacity, useful when different services have very different throughput characteristics), or at the client SDK level (a well-behaved client self-throttles before even sending a request, reducing wasted network round-trips on requests that would just get rejected). In a distributed system, the counters that track usage need to be shared across all gateway or service instances — typically backed by Redis with atomic INCR and EXPIRE commands — otherwise each instance enforces its own independent limit and the effective global limit becomes (per-instance limit x number of instances), defeating the purpose.

🏏

Cricket analogy: Ground security checks tickets at the outer perimeter gate (API gateway), stand-specific stewards check seating capacity within each stand (service level), and well-organized tour groups self-pace their arrival times (client SDK) to avoid all converging on the gate at once.

python

# Token bucket rate limiter backed by Redis, safe across multiple gateway instances
import time
import redis

r = redis.Redis(host="redis", port=6379, decode_responses=True)

def allow_request(client_id: str, capacity: int = 100, refill_per_sec: int = 10) -> bool:
    now = time.time()
    key = f"ratelimit:{client_id}"

    pipe = r.pipeline()
    pipe.hmget(key, "tokens", "last_refill")
    tokens, last_refill = pipe.execute()[0]

    tokens = float(tokens) if tokens else capacity
    last_refill = float(last_refill) if last_refill else now

    elapsed = now - last_refill
    tokens = min(capacity, tokens + elapsed * refill_per_sec)

    if tokens < 1:
        return False

    tokens -= 1
    r.hmset(key, {"tokens": tokens, "last_refill": now})
    r.expire(key, 3600)
    return True

Fixed window counters are simple but have a boundary flaw: a client can send its full quota right at the end of one window and its full quota again right at the start of the next, effectively doubling the intended rate for a brief period. Prefer a sliding window or token bucket algorithm if bursty edge-timing abuse is a real concern for your API.

Always return a clear, standardized response when a request is throttled: HTTP 429 Too Many Requests along with a Retry-After header telling the client exactly how long to wait. This lets well-behaved clients back off automatically instead of hammering the endpoint immediately again.

Rate limiting protects services from being overwhelmed by internal callers as well as external clients, enabling predictable capacity planning.
Token bucket allows controlled bursts up to a capacity while enforcing a steady average refill rate.
Sliding window algorithms avoid the boundary problem of fixed windows, where clients can double their effective rate around window edges.
Rate limits can be enforced at the API gateway, at individual services, and client-side via SDK self-throttling.
Distributed rate limiting requires a shared counter store (commonly Redis) so limits are enforced globally, not per-instance.
Throttled requests should return HTTP 429 with a Retry-After header so clients can back off intelligently.
Per-tenant rate limits are a key tool for preventing noisy-neighbor effects in multi-tenant systems.