How Do You Mitigate Hot Spots in a Sharded System?
Learn how to detect and fix hot spots in sharded systems with key salting, caching, partition splitting, and rate limiting.
Expected Interview Answer
A hot spot is a single shard, partition, or key that receives disproportionately more traffic than its peers, and it is mitigated by spreading that skewed load — through better key design (salting/prefixing hot keys), splitting a hot partition further, caching hot reads in front of storage, or applying request-level rate limiting so one hot key cannot starve the rest of the cluster.
Hot spots typically arise from skewed access patterns: a celebrity user’s profile, a viral product ID, or a monotonically increasing key (like a timestamp-based primary key) that funnels all new writes onto the single shard currently owning the newest range. The first line of defense is key design — salting a hot key with a random or hashed prefix spreads its writes across multiple shards, at the cost of needing a scatter-gather read to reassemble results. For read-heavy hot spots, an in-memory cache (Redis, CDN edge cache) in front of the database absorbs the bulk of traffic so the hot shard only serves cache misses. For write-heavy hot spots, some systems detect a hot partition automatically and split it further (as DynamoDB and Bigtable do), or route through a write-behind buffer that batches and smooths bursts. Finally, per-key rate limiting and circuit breakers prevent one runaway hot key from exhausting shared resources and degrading the whole cluster for unrelated keys.
- Prevents one overloaded shard from becoming a bottleneck or single point of failure for the whole cluster
- Key salting spreads writes evenly, avoiding monotonic-key hotspotting on the newest shard
- Caching hot reads dramatically reduces load reaching the underlying storage layer
- Per-key rate limiting isolates a runaway hot key so it does not degrade service for unrelated keys
AI Mentor Explanation
A hot spot is like every single delivery in an over being aimed at the same fielder because batters keep hitting to that one spot on the boundary, exhausting them while the rest of the field stands idle. The fix is like a captain rotating extra cover into that zone, or coaching the bowler to vary line and length so shots spread across the whole field instead of piling onto one fielder. Similarly, in a sharded system, a hot key or partition gets extra caching capacity or its traffic gets spread across more shards so no single one bears all the load. That deliberate load-spreading is exactly how hot spot mitigation works.
Step-by-Step Explanation
Step 1
Detect the hot key or shard
Monitor per-shard/per-key request rates to identify traffic that is far above the average, before it saturates a node.
Step 2
Redesign the key for write hot spots
Salt or hash-prefix a monotonic or celebrity key so its writes spread across multiple shards instead of one.
Step 3
Cache in front for read hot spots
Put a Redis or CDN edge cache in front of the hot key so repeated reads are absorbed before reaching storage.
Step 4
Split or isolate as a last resort
Split the hot partition into finer-grained ranges, or apply per-key rate limiting/circuit breaking so it cannot starve the rest of the cluster.
What Interviewer Expects
- Correctly identifies common causes of hot spots: celebrity keys, monotonic keys, viral content
- Distinguishes mitigation for read-heavy hot spots (caching) versus write-heavy hot spots (key salting/splitting)
- Mentions real system behaviors like DynamoDB/Bigtable automatic partition splitting
- Discusses the trade-off of salting: writes spread out but reads need scatter-gather to reassemble
Common Mistakes
- Proposing “just add more shards” without addressing that a monotonic key still funnels onto the newest shard
- Not distinguishing between read hot spots (fixed by caching) and write hot spots (fixed by key design)
- Ignoring the read-side cost of salting (needing to query multiple shards and merge results)
- Forgetting that per-key rate limiting/circuit breaking protects the rest of the cluster even before a permanent fix ships
Best Answer (HR Friendly)
“A hot spot happens when one piece of data, like a viral product or a celebrity’s profile, gets way more traffic than everything else, overloading whichever single server or shard holds it. You fix it by spreading that data’s load out — caching it so most requests never hit the database, splitting it across more servers, or redesigning the key so new data does not all pile onto the same place.”
Code Example
import random
SALT_BUCKETS = 10
def write_key(base_key: str, value):
# Spread writes for a hot key across N sub-keys / shards
salt = random.randint(0, SALT_BUCKETS - 1)
salted_key = f"{base_key}#{salt}"
shard = get_shard_for_key(salted_key)
shard.write(salted_key, value)
def read_aggregate(base_key: str):
# Scatter-gather: read from every salt bucket and merge
results = []
for salt in range(SALT_BUCKETS):
salted_key = f"{base_key}#{salt}"
shard = get_shard_for_key(salted_key)
results.extend(shard.read(salted_key))
return merge(results)Follow-up Questions
- How does DynamoDB or Bigtable automatically detect and split a hot partition?
- What is the trade-off between key salting granularity (few vs many salt buckets) and read cost?
- How would you handle a hot spot caused by a suddenly viral piece of content in real time?
- How does consistent hashing with virtual nodes help or not help with hot spots caused by data skew rather than server count?
MCQ Practice
1. What commonly causes write hot spots in a sharded database using a timestamp-based primary key?
A monotonically increasing key means every new write lands in the same range, overloading the shard that owns the latest values.
2. What is the main cost of salting a hot key to spread its writes across shards?
Spreading a key across multiple sub-keys means reconstructing a full read requires querying every bucket and combining the results.
3. What is the best mitigation for a read-heavy hot spot on a single popular key?
Caching absorbs the bulk of repeated read traffic so the underlying hot shard only needs to serve cache misses.
Flash Cards
What is a hot spot? — A single shard, partition, or key receiving disproportionately more traffic than its peers.
Common cause of write hot spots? — A monotonically increasing key (like a timestamp) that funnels all new writes onto one shard.
Fix for read hot spots? — Cache the hot key in front of storage (Redis or CDN) to absorb repeated reads.
Fix for write hot spots? — Salt/hash-prefix the key to spread writes across shards, or split the hot partition further.