Rate Limiting
Rate limiting is a technique for controlling the number of requests a client can make to a service within a given time window, used to protect systems from overload, abuse, and unfair resource consumption.
Definition
Rate limiting is a technique for controlling the number of requests a client can make to a service within a given time window, used to protect systems from overload, abuse, and unfair resource consumption.
Overview
Rate limiting enforces a cap on request volume — for example, 100 requests per minute per API key — and rejects or delays requests once that cap is exceeded, usually returning an HTTP 429 'Too Many Requests' response. It is commonly enforced at an API Gateway or Load Balancer so the limit is applied consistently before traffic ever reaches backend services. Several algorithms are used in practice. The token bucket and leaky bucket algorithms smooth out bursts of traffic over time, allowing some burstiness while enforcing an average rate. Fixed-window counters are simple but can allow bursts at window boundaries, while sliding-window approaches correct for that at the cost of slightly more bookkeeping. The right choice depends on how tolerant the system is of short bursts versus needing a strictly even request rate. Rate limiting serves several purposes at once: protecting infrastructure from being overwhelmed (including as a defense against DDoS Attack traffic), ensuring fair usage across many tenants of a shared API, and enforcing business rules like free-tier quotas on a paid API product. It's a close cousin of the Circuit Breaker Pattern — both protect system health under stress — but rate limiting caps client-driven demand proactively, while a circuit breaker reacts to observed downstream failures.
Key Concepts
- Caps request volume per client, API key, IP address, or user within a time window
- Common algorithms: token bucket, leaky bucket, fixed window, and sliding window
- Typically enforced at the API gateway or load balancer layer
- Returns HTTP 429 responses with retry guidance when limits are exceeded
- Protects backend services from overload and abuse
- Enables tiered usage plans (e.g., free vs. paid API quotas)
- Helps mitigate certain classes of denial-of-service traffic
Use Cases
Frequently Asked Questions
From the Blog
What Is a Learning Rate and How to Tune It
The learning rate controls how big a step a model takes when updating its weights during training — the single most important hyperparameter to get right.
Read More AI & TechnologyWhat Is CTR? Click-Through Rate Explained
CTR, or click-through rate, measures the percentage of people who click a link, ad, or search result out of everyone who saw it. This guide explains how it's calculated, what affects it, and how to improve it.
Read More AI & TechnologyHow to Choose Learning Rate and Epochs for a LoRA Run
Start from a conservative configuration, run a short training pass, and let the loss curves tell you what to change. Rank, alpha, learning rate and epoch count interact, so the productive method is one variable at a time against a held-out set rather than a search over everything at once.
Read More AI & TechnologyHow to Measure Hallucination Rate Without Manual Review
You can score factuality automatically by splitting each answer into atomic claims and checking every claim against the retrieved source text. This article shows how to build that pipeline, how to calibrate its verdicts against a small human-labelled sample, and where it quietly fails.
Read More