Latency vs Throughput: What Is the Trade-off?
Learn the difference between latency and throughput, why batching trades one for the other, and how Little’s Law explains the balance.
Expected Interview Answer
Latency is the time a single request takes from start to finish, while throughput is the number of requests a system completes per unit of time, and optimizing hard for one often costs the other because batching, queuing, and concurrency controls that raise throughput add waiting time to any individual request.
A system tuned purely for low latency processes each request the instant it arrives, using dedicated resources and minimal batching, which keeps individual response times short but wastes capacity when load is uneven. A system tuned purely for high throughput batches work, fills queues, and runs requests concurrently across all available resources, driving up total completed work per second but making each request wait longer inside a queue or a batch window before it is served. Little’s Law formalizes this: average number of requests in the system equals arrival rate multiplied by average time in system, so as concurrency (and throughput) rises without adding capacity, queueing delay and therefore latency rises too. Engineers usually target a latency percentile budget (p50/p95/p99) alongside a throughput floor, then tune batch sizes, connection pools, and concurrency limits until both meet the service level objective rather than maximizing either one alone.
- Clarifies why maximizing raw throughput can silently blow tail latency (p99) budgets
- Explains why batching and queuing help throughput but hurt per-request latency
- Gives a shared vocabulary (Little’s Law) for reasoning about concurrency vs delay
- Guides capacity planning toward a latency SLO plus a throughput floor, not one metric alone
AI Mentor Explanation
Latency is how fast a single delivery travels from the bowler’s hand to the keeper’s gloves, while throughput is how many overs the whole team bowls in a day. A captain chasing throughput packs the schedule with back-to-back overs and short breaks between bowlers, which gets more overs bowled but means any one bowler sometimes waits pad-up time before their over starts. Chasing pure speed on every single ball, by contrast, tires bowlers out and slows the over rate down later in the day. The best sides balance both: enough over-rate discipline to finish the day’s quota while still keeping each individual delivery quick.
Step-by-Step Explanation
Step 1
Define both metrics precisely
Latency is measured per request (often as p50/p95/p99 percentiles); throughput is measured as requests completed per second across the whole system.
Step 2
Identify the shared resource
Batching, connection pooling, and queuing raise throughput by sharing fixed resources across more requests, which is exactly what adds queueing delay to each request.
Step 3
Apply Little’s Law
Average requests in the system = arrival rate x average time in system, so pushing concurrency and throughput up without adding capacity increases latency.
Step 4
Set a joint SLO
Pick a target throughput floor and a tail latency ceiling (e.g., p99 < 200ms at 5,000 req/s), then tune batch size and concurrency limits to satisfy both.
What Interviewer Expects
- Correctly distinguishes per-request latency from system-wide throughput
- Explains why batching/queuing raises throughput at the cost of individual request latency
- References Little’s Law or an equivalent queueing intuition
- Proposes tuning toward a joint SLO (latency percentile + throughput floor) rather than maximizing one metric
Common Mistakes
- Treating latency and throughput as the same metric or assuming one always implies the other
- Optimizing average latency while ignoring tail latency (p99) which users actually notice
- Forgetting that adding concurrency without adding capacity increases queueing delay
- Proposing “just add more batching” without acknowledging the added per-request delay
Best Answer (HR Friendly)
“Latency is how long one single task takes to finish, and throughput is how many tasks the whole system gets done in total. If you batch things together to get more done overall, each individual task usually has to wait a bit longer, so improving one often costs a little of the other, and the goal is to find a balance that keeps both fast enough for what users actually need.”
Code Example
class BatchingQueue {
constructor(batchSize = 50, maxWaitMs = 10) {
this.batchSize = batchSize
this.maxWaitMs = maxWaitMs
this.pending = []
this.timer = null
}
enqueue(request) {
return new Promise((resolve) => {
this.pending.push({ request, resolve })
if (this.pending.length >= this.batchSize) {
this.flush() // throughput win: full batch, no extra wait
} else if (!this.timer) {
// latency trade-off: cap how long a lone request waits for a batch
this.timer = setTimeout(() => this.flush(), this.maxWaitMs)
}
})
}
async flush() {
clearTimeout(this.timer)
this.timer = null
const batch = this.pending.splice(0, this.pending.length)
if (batch.length === 0) return
const results = await processBatch(batch.map((b) => b.request))
batch.forEach((b, i) => b.resolve(results[i]))
}
}Follow-up Questions
- How does Little’s Law relate arrival rate, latency, and concurrency?
- Why can average latency look fine while p99 latency is terrible under high throughput?
- How would you design an API that needs both low latency and high throughput for different endpoints?
- What role does connection pooling play in this trade-off?
MCQ Practice
1. Which technique typically increases throughput at the cost of per-request latency?
Batching shares processing across more requests, raising overall throughput but adding wait time inside the batch for each individual request.
2. What does Little’s Law state?
Little’s Law (L = λW) links the number of requests in flight to arrival rate and time spent in the system, explaining why higher concurrency raises latency without more capacity.
3. Why is p99 latency often a better SLO target than average latency?
Average latency can look healthy while a meaningful fraction of requests (the tail) are slow; p99 exposes that tail directly.
Flash Cards
Latency vs throughput? — Latency is time for one request to finish; throughput is total requests completed per second across the system.
Why does batching raise throughput but hurt latency? — Batching shares fixed resources across more requests at once, but each request must wait for the batch to fill or flush.
Little’s Law? — Average requests in system = arrival rate x average time in system; higher concurrency without more capacity raises latency.
What should you target instead of one metric alone? — A joint SLO: a tail latency ceiling (e.g., p99) plus a throughput floor, tuned together.