What is a Kafka partition and why is it the unit of parallelism?
Learn what a Kafka partition is, how offsets and keys work, why ordering is per-partition, and why partitions are Kafka's unit of parallelism, with examples.
Expected Interview Answer
A Kafka partition is an ordered, append-only log that is one slice of a topic; it is the unit of parallelism because each partition can be written, stored, and consumed independently, and within a consumer group exactly one consumer reads a given partition at a time.
A topic is split into one or more partitions, and each message lands in exactly one partition where it gets a monotonically increasing offset. Ordering is guaranteed only within a partition, never across the whole topic. Because partitions are independent, throughput scales by adding partitions and consumers, but the number of partitions caps how many consumers in a group can work in parallel — extra consumers sit idle.
- Enables horizontal scaling of both producers and consumers
- Guarantees strict ordering within each partition
- Spreads storage and load across multiple brokers
- Lets a consumer group divide work by assigning partitions
- Provides a natural key-based routing and locality mechanism
AI Mentor Explanation
Think of a topic as a full match and each partition as one batter's individual scoring log. Every ball that batter faces is appended in strict order to their own log, and one dedicated scorer tracks only that batter. You get more scorers working at once by having more batters logged separately, but you can never have two scorers on the same batter's sheet without confusion.
Step-by-Step Explanation
Step 1
A topic is divided into partitions
When you create a topic you choose a partition count; each partition is an independent append-only log stored on a broker.
Step 2
Messages get an offset
Every record written to a partition receives a strictly increasing offset that identifies its position and preserves order within that partition.
Step 3
A key decides the partition
Producers hash the record key to pick a partition, so all records sharing a key land in the same partition and keep their relative order.
Step 4
Consumers split partitions in a group
Kafka assigns each partition to exactly one consumer in a group, so parallelism is bounded by the partition count.
Step 5
Scale by adding partitions
More partitions allow more consumers to read concurrently, but ordering guarantees never extend beyond a single partition.
What Interviewer Expects
- Partition defined as an ordered, append-only log with offsets
- Ordering guaranteed only within a partition, not across a topic
- One consumer per partition within a consumer group
- Partition count as the ceiling on consumer-group parallelism
- How message keys route records to partitions
Common Mistakes
- Claiming Kafka guarantees ordering across an entire topic
- Thinking more consumers than partitions increases throughput
- Confusing partitions with replicas
- Forgetting that a null key spreads records round-robin
- Assuming partition count can be reduced later without consequences
Best Answer (HR Friendly)
“A Kafka partition is one ordered slice of a topic, like a single checkout lane in a store. Because each lane works on its own, you scale by adding lanes and staff, but one lane is served by one worker at a time, which is why partitions decide how much work can happen in parallel.”
Code Example
# Six partitions lets up to six consumers in a group read in parallel
kafka-topics.sh --create \
--topic orders \
--partitions 6 \
--replication-factor 3 \
--bootstrap-server localhost:9092
# Inspect how partitions map to brokers
kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092Follow-up Questions
- How does the producer decide which partition a record goes to?
- What happens when you have more consumers than partitions in a group?
- Why can you increase but not easily decrease partition count?
- How do partitions relate to ordering guarantees?
- What is the trade-off of having too many partitions in a cluster?
MCQ Practice
1. Within a single consumer group, how many consumers can read one partition at a time?
Kafka assigns each partition to exactly one consumer in a group, which is why partition count bounds group parallelism.
2. Kafka guarantees message ordering at what scope?
Ordering is guaranteed only within a partition; there is no global ordering across a topic's partitions.
3. What primarily determines which partition a keyed record lands in?
The default partitioner hashes the record key, so identical keys route to the same partition and preserve their order.
Flash Cards
What is a Kafka partition? — An ordered, append-only log that is one slice of a topic, with each record identified by an increasing offset.
Where is ordering guaranteed? — Only within a single partition, never across a whole topic.
Why is a partition the unit of parallelism? — Each partition is read by exactly one consumer per group, so partition count caps concurrency.
What routes a record to a partition? — A hash of the record key; a null key spreads records round-robin.