How Do You Design a Good Partition Key?
Learn how to design partition keys that avoid hot partitions, align with access patterns, and scale writes across a distributed store.
Expected Interview Answer
A good partition key spreads data and traffic evenly across all partitions (avoiding hot spots) while still keeping any data that must be read or ordered together — like one entity’s related records — on the same partition, which usually means choosing a high-cardinality, access-pattern-aligned key rather than a low-cardinality or monotonically increasing one.
A low-cardinality key (like “status” with only three possible values) forces all traffic into a handful of partitions no matter how many partitions exist, creating hot spots that bottleneck throughput. A monotonically increasing key (like an auto-incrementing order ID or a raw timestamp) causes every new write to land on the same “latest” partition, another classic hot-spot pattern seen in systems like DynamoDB and Bigtable. The fix is usually a composite or high-cardinality key — for example, using customer ID (or a hash of it) instead of order creation timestamp, or adding a random suffix / shard-count prefix ("write sharding") to spread hot monotonic keys across N partitions while still being able to reconstruct order via a secondary index or scatter-gather read. The right key ultimately depends on the read access pattern: whichever field is used to fetch data together should generally be (or be part of) the partition key, since cross-partition queries are expensive scatter-gather operations.
- Even key cardinality prevents hot partitions and keeps throughput scaling linearly with partition count
- Aligning the key with the dominant read pattern avoids expensive cross-partition scatter-gather queries
- Write-sharding techniques fix monotonic-key hot spots without losing the ability to query by time
- Good key design is usually cheaper to get right upfront than to migrate later at scale
AI Mentor Explanation
Partition key design is like deciding how to split practice nets among coaches — assigning every batter to the single net closest to the pavilion door creates a crowded bottleneck, while assigning by batter surname initial spreads players evenly across all available nets. A bad key is like grouping by “batting style” when ninety percent of the squad bats right-handed — nearly everyone still piles into one net. A good key, like a rotating roster number, spreads players evenly and still lets a coach quickly find all of one batter’s sessions together. That balance between even spread and keeping related sessions together is exactly what a good partition key achieves.
Step-by-Step Explanation
Step 1
Identify the dominant access pattern
Determine which field is used most often to fetch or group data together, since that field should drive the key.
Step 2
Check cardinality
Ensure the candidate key has enough distinct values relative to the number of partitions to avoid a handful of partitions absorbing most traffic.
Step 3
Watch for monotonic hot spots
Reject keys like raw auto-increment IDs or timestamps that always point new writes at the same “latest” partition.
Step 4
Apply write-sharding if needed
For inherently hot or monotonic keys, add a random or bucketed prefix/suffix to spread writes, then use a secondary index or scatter-gather to preserve queryability.
What Interviewer Expects
- Explains hot partitions and why low-cardinality or monotonic keys cause them
- Connects key choice to the dominant read/query access pattern
- Names a concrete mitigation: composite key, hashing, or write-sharding with a secondary index
- Gives a realistic example (e.g., DynamoDB/Bigtable hot partition from timestamp keys)
Common Mistakes
- Choosing a partition key purely for uniqueness without considering access patterns
- Using an auto-increment ID or raw timestamp as the partition key at write-heavy scale
- Not recognizing when cross-partition scatter-gather queries become a performance problem
- Ignoring that write-sharding trades off read complexity to fix a hot-write problem
Best Answer (HR Friendly)
“A good partition key spreads data evenly across all the machines storing it so no single machine becomes a bottleneck, while still keeping data that is usually needed together — like one customer’s records — on the same machine so it is fast to fetch. Picking something like a raw timestamp as the key is a classic mistake because every new write piles onto the same “latest” partition.”
Code Example
import random
SHARD_COUNT = 10
# BAD: raw timestamp as partition key -> every write today lands on
# whichever partition owns “now”, creating a hot partition.
def bad_partition_key(event):
return event.created_at.strftime("%Y-%m-%d")
# GOOD: bucket the hot timestamp key with a random shard suffix,
# spreading writes across SHARD_COUNT partitions.
def sharded_partition_key(event):
date_prefix = event.created_at.strftime("%Y-%m-%d")
shard = random.randint(0, SHARD_COUNT - 1)
return f"{date_prefix}#shard{shard}"
# To read “all events for today” back in order, fan out a query
# across all SHARD_COUNT keys and merge results (scatter-gather read).
def read_events_for_date(date_str):
results = []
for shard in range(SHARD_COUNT):
results.extend(query_partition(f"{date_str}#shard{shard}"))
return sorted(results, key=lambda e: e.created_at)Follow-up Questions
- How would you detect a hot partition in production before it causes an outage?
- What is write-sharding and what does it cost you on the read path?
- Why does DynamoDB recommend high-cardinality partition keys and what happens if you ignore that?
- How would you re-partition a table without downtime once you discover a bad key choice?
MCQ Practice
1. Why does using a raw auto-incrementing ID as a partition key often cause problems at scale?
A monotonically increasing key concentrates all new writes onto whichever partition currently owns the highest range, overloading that one partition.
2. What should primarily drive the choice of a partition key?
A good partition key aligns with how data is queried so related records stay on the same partition, avoiding expensive cross-partition scatter-gather reads.
3. What is “write-sharding” used to solve?
Write-sharding adds a random or bucketed component to an otherwise hot key so writes spread across many partitions instead of piling onto one.
Flash Cards
Hot partition? — A partition receiving disproportionate traffic because the partition key has low cardinality or is monotonically increasing.
What should drive partition key choice? — The dominant read/access pattern — data queried together should share a partition key.
Write-sharding? — Adding a random or bucketed suffix/prefix to a hot key to spread writes across partitions, at the cost of needing scatter-gather reads.
Classic bad partition key example? — A raw timestamp or auto-increment ID, which piles all new writes onto the same “latest” partition.