How do you scale Kafka throughput?
Learn how to scale Kafka throughput with partitions, consumer groups, broker scaling, and producer batching, plus interview answers and examples.
Expected Interview Answer
You scale Kafka throughput mainly by increasing the number of partitions on a topic and adding consumers to a consumer group, so producers and consumers can work in parallel across brokers instead of being bottlenecked on a single partition.
Throughput grows when work is spread across more partitions, more brokers, and more consumer instances, since one partition is only ever read by one consumer within a group. Beyond partitioning, you tune producer batching (batch.size, linger.ms), enable compression, size replication and acks appropriately, add brokers to distribute load, and ensure downstream consumers process fast enough to keep lag low. Hardware — disks, network, and page cache — and even message keying strategy all shape the ceiling.
- Parallelism across partitions and brokers
- Higher aggregate write and read rates
- Horizontal scaling by adding brokers and consumers
- Lower consumer lag under heavy load
- Tunable trade-off between latency and throughput
AI Mentor Explanation
Think of a single scorer trying to record every ball of a packed tournament alone — a hopeless bottleneck. Assign one scorer per match running in parallel and total capacity multiplies. Kafka partitions work the same way: each partition is a match with its own scorer (consumer), so adding partitions and consumers lets far more deliveries be recorded per minute across the whole ground.
Step-by-Step Explanation
Step 1
Partition the topic
Increase partition count so producers and consumers can work in parallel; one partition is read by only one consumer per group, so partitions cap consumer parallelism.
Step 2
Scale the consumer group
Add consumer instances up to the partition count so each partition is drained by a dedicated worker and lag stays low.
Step 3
Tune the producer
Raise batch.size and linger.ms, enable compression (lz4/zstd), and pick an acks level that balances durability against throughput.
Step 4
Add brokers
Expand the cluster and rebalance partitions to spread disk, network, and CPU load across more machines.
Step 5
Optimize the downstream
Make consumer processing fast and idempotent, commit offsets efficiently, and monitor consumer lag to find the real bottleneck.
What Interviewer Expects
- Understanding that partitions drive parallelism
- One-consumer-per-partition rule within a group
- Producer batching and compression tuning
- Role of brokers and replication in scaling
- Using consumer lag as the key throughput metric
Common Mistakes
- Adding more consumers than partitions and expecting extra parallelism
- Ignoring producer batching and sending one record per request
- Assuming more replication automatically means more throughput
- Forgetting the downstream consumer is often the real bottleneck
- Over-partitioning and inflating metadata and rebalance overhead
Best Answer (HR Friendly)
“You scale Kafka by splitting each topic into more partitions so many workers can read and write at the same time, then adding servers and consumers to share the load. You also tune how messages are batched and compressed, and watch how far behind consumers are so nothing gets stuck.”
Code Example
# Add partitions to raise parallelism (cannot be decreased later)
kafka-topics.sh --bootstrap-server localhost:9092 \
--alter --topic orders --partitions 12
# Producer config for higher throughput
batch.size=65536 # larger batches
linger.ms=10 # wait briefly to fill batches
compression.type=lz4 # cheap, fast compression
acks=1 # balance durability vs speedFollow-up Questions
- Why can't you have more active consumers than partitions in a group?
- How do linger.ms and batch.size trade latency for throughput?
- What happens to ordering when you increase partition count?
- How does consumer lag help you locate a bottleneck?
- When would compression hurt rather than help throughput?
MCQ Practice
1. What most directly limits how many consumers in a group can read a topic in parallel?
Within a consumer group each partition is assigned to exactly one consumer, so partition count caps active consumer parallelism.
2. Which producer setting encourages larger batches at the cost of a little latency?
linger.ms makes the producer wait briefly to accumulate more records into a batch, improving throughput at a small latency cost.
3. You added 20 consumers to a group but a topic has only 8 partitions. What happens?
Only 8 consumers can be assigned partitions; the remaining 12 sit idle as standby until a partition frees up.
Flash Cards
What caps consumer parallelism in a group? — The partition count — one partition is consumed by only one consumer per group.
Two key producer throughput knobs? — batch.size (bigger batches) and linger.ms (wait to fill batches), often with compression.
How do you scale reads and writes horizontally? — Add partitions, add brokers, and add consumer instances so load spreads across the cluster.
Best metric for spotting a throughput bottleneck? — Consumer lag — rising lag shows consumers can't keep up with produced volume.