How Do You Rebalance Shards Without Downtime?
Learn how to rebalance database shards live using dual-write, backfill, and verified cutover with zero downtime.
Expected Interview Answer
Rebalancing shards without downtime means gradually migrating a subset of data to new or less-loaded shards while keeping both the old and new locations correctly serving reads and writes, typically using a dual-write or replicate-then-cutover approach with a routing layer that flips ownership only after the migrated data is verified consistent.
The core technique is to pick the keys or ranges that need to move, start replicating their writes to the destination shard while the source shard keeps serving as the source of truth, and backfill historical data in the background. Once the destination is caught up and verified, the routing layer atomically flips which shard owns those keys, and only after a safety window with no more reads hitting the old location is the source data deleted. Using consistent hashing with virtual nodes minimizes how much data needs to move for any single rebalance, and having an abstraction layer between application code and physical shard placement (a routing table or proxy) means clients never need to know a migration happened. Throttling the migration’s throughput protects live traffic from being starved by the backfill process.
- Keeps the system serving reads and writes throughout the entire migration
- Consistent hashing minimizes the fraction of data that must move per rebalance
- Dual-write plus verified cutover avoids losing or duplicating data mid-migration
- A routing abstraction keeps rebalancing invisible to application clients
AI Mentor Explanation
Rebalancing shards is like moving part of an overcrowded stand’s ticket holders to a newly opened stand mid-match without ever stopping play. Stewards start directing new arrivals with matching tickets to the new stand while the old stand keeps serving everyone already seated there, and only once the new stand is fully staffed and verified does the gate officially switch which stand a ticket range belongs to. Nobody loses their seat and nobody is double-counted because the switch happens atomically after verification. That live, verified hand-off is exactly how a database rebalances shards without downtime.
Step-by-Step Explanation
Step 1
Select the migration set
Identify which key ranges or virtual nodes are moving, chosen to minimize total data movement (e.g. via consistent hashing).
Step 2
Start dual-write and backfill
New writes for the migrating keys are mirrored to the destination shard while historical data is copied in the background.
Step 3
Verify consistency
Compare source and destination checksums or row counts to confirm the destination is fully caught up before cutover.
Step 4
Atomically flip routing and clean up
Update the routing table to point at the new shard, then after a safety window delete the now-redundant data from the source.
What Interviewer Expects
- Describes a replicate-then-verify-then-cutover pattern, not a stop-the-world copy
- Mentions a routing layer or metadata service that abstracts shard placement from clients
- Recognizes consistent hashing reduces how much data must move per rebalance
- Discusses throttling migration throughput to protect live production traffic
Common Mistakes
- Proposing to simply stop writes during the entire migration window
- Forgetting to verify data consistency before flipping ownership
- Not considering how clients discover the new shard location (routing/metadata layer)
- Ignoring the need to throttle backfill so it does not starve live traffic
Best Answer (HR Friendly)
“Rebalancing shards without downtime means moving a slice of data to a new location while the system keeps running normally. You start copying the data in the background and mirroring new writes to both the old and new spot, then once you’ve confirmed the new spot is fully caught up and correct, you flip a switch so all future requests go there instead. Because that switch only happens after verification, users never notice the move happening.”
Code Example
def migrate_shard_range(key_range, source, dest, router):
backfill(source, dest, key_range) # copy historical rows
router.enable_dual_write(key_range, dest) # mirror new writes to dest too
while not is_caught_up(source, dest, key_range):
sleep(POLL_INTERVAL)
if not verify_consistency(source, dest, key_range):
raise MigrationError("checksum mismatch, aborting cutover")
router.flip_ownership(key_range, dest) # atomic: reads now go to dest
wait(SAFETY_WINDOW_SECONDS)
router.disable_dual_write(key_range)
delete_range(source, key_range) # reclaim space on old shard
Follow-up Questions
- How would you handle a write that arrives mid-migration for a key that has already been cut over?
- What happens if the destination shard falls behind during backfill on a high-write-volume table?
- How does consistent hashing with virtual nodes reduce the blast radius of a rebalance?
- How would you roll back a rebalance if verification fails after cutover?
MCQ Practice
1. What is the safest general pattern for rebalancing shards in a live system?
Mirroring writes while backfilling, verifying consistency, and only then flipping ownership avoids downtime and data loss.
2. Why is a routing/metadata layer important during shard rebalancing?
A routing layer lets shard ownership change transparently to application code, which just asks the router where a key lives.
3. Why throttle the backfill process during a shard migration?
Uncapped backfill throughput can starve live queries of resources, so migrations are rate-limited to protect production traffic.
Flash Cards
Zero-downtime rebalance pattern? — Dual-write plus background backfill, verify consistency, then atomically flip routing ownership.
Why use consistent hashing for rebalancing? — It minimizes the fraction of keys that must move when capacity is added or removed.
Role of the routing layer? — Abstracts physical shard placement so clients are unaffected when data migrates between shards.
Why throttle backfill? — To prevent the migration from starving live production traffic of I/O and bandwidth.