How to Design a File Sync Service
Design a Dropbox-style file sync service: chunking, content-hash dedup, metadata versioning, and conflict resolution.
Expected Interview Answer
A file sync service (like Dropbox) splits files into fixed-size chunks, uploads only changed chunks to block storage identified by content hash, and keeps a metadata service that tracks file-to-chunk mappings and version history so every client device can detect and download exactly what changed.
On the client, a watcher detects file changes and splits the file into content-defined chunks (typically 4MB), hashing each chunk so identical chunks across files or versions are deduplicated. Only chunks not already present in block storage are uploaded; the client then commits a new file version to a metadata service that records the ordered list of chunk hashes composing that version. Other devices subscribed to the same account receive a notification (via a long-lived connection or push) that a file changed, fetch the updated chunk list from metadata, diff it against their local chunk list, and download only the missing chunks before reassembling the file locally. Conflict handling matters when two devices edit offline simultaneously — the service detects a version conflict via a version vector or last-known-version check and either creates a conflicted copy or applies operational-transform/CRDT-style merging depending on file type.
- Chunk-level deduplication minimizes bandwidth and storage for small edits to large files
- Content-hash addressing lets identical chunks be shared across files and versions automatically
- Metadata/version tracking lets every device sync incrementally instead of re-downloading whole files
- Version conflict detection prevents silent data loss when multiple devices edit offline
AI Mentor Explanation
A file sync service is like updating a match scorecard shared across every broadcaster's truck without resending the whole card each time. Only the specific over that changed (a chunk) gets transmitted, identified by its unique over number (content hash), while unchanged overs already held by each truck are skipped entirely. Every truck then reassembles its local copy of the full scorecard from the pieces it already has plus the newly sent over. If two trucks record conflicting updates to the same over while briefly disconnected, the system flags a conflict rather than silently overwriting one. That chunk-level, hash-identified update mechanism is exactly how a file sync service minimizes data transfer.
Step-by-Step Explanation
Step 1
Detect and chunk changes
A client watcher detects a file change and splits the file into content-hashed chunks.
Step 2
Upload only new chunks
The client checks which chunk hashes are missing from block storage and uploads only those.
Step 3
Commit a new version to metadata
The metadata service records the ordered chunk list representing the new file version.
Step 4
Notify and sync other devices
Other devices are notified, diff their local chunk list against the new one, download missing chunks, and reassemble the file.
What Interviewer Expects
- Explains chunking and content-hash addressing as the basis for deduplication
- Separates block storage (chunk bytes) from a metadata service (chunk-to-file mapping and versions)
- Discusses incremental sync — only transferring what changed, not the whole file
- Addresses conflict detection when two devices edit the same file offline
Common Mistakes
- Describing whole-file upload/download instead of chunk-level incremental sync
- Ignoring deduplication opportunities across files with shared content
- Not addressing offline edit conflicts between multiple devices
- Conflating the metadata service with the block storage layer
Best Answer (HR Friendly)
“A file sync service breaks files into small pieces and only uploads the pieces that actually changed, rather than the whole file every time. It keeps track of which pieces make up each version of a file, so other devices can figure out exactly what they are missing and download just that. If two devices change the same part of a file while offline, it detects the conflict instead of quietly losing someone's changes.”
Code Example
def sync_file(local_path, remote_metadata):
chunks = chunk_file(local_path, chunk_size=4 * 1024 * 1024)
chunk_hashes = [sha256(chunk) for chunk in chunks]
missing = [h for h in chunk_hashes if not block_storage.exists(h)]
for h, chunk in zip(chunk_hashes, chunks):
if h in missing:
block_storage.upload(h, chunk)
new_version = {
"file_id": remote_metadata.file_id,
"chunk_hashes": chunk_hashes,
"parent_version": remote_metadata.current_version,
}
if remote_metadata.current_version != new_version["parent_version"]:
return handle_conflict(local_path, remote_metadata)
metadata_service.commit_version(new_version)Follow-up Questions
- How would you handle a device syncing after being offline for weeks with thousands of changed files?
- What chunk size trade-offs exist between deduplication granularity and metadata overhead?
- How would you detect and resolve conflicting edits made on two devices while offline?
- How would you design bandwidth-efficient sync for a mobile client on a metered connection?
MCQ Practice
1. Why do file sync services split files into content-hashed chunks?
Chunking with content hashing lets the system detect exactly which pieces changed and reuse identical chunks across files.
2. What is the role of the metadata service in a file sync architecture?
The metadata service maps files and versions to their ordered chunk lists, separate from the block storage holding chunk bytes.
3. What typically happens when two devices edit the same file offline and then reconnect?
Version conflict detection (via version vectors or last-known-version checks) prevents silent data loss between offline edits.
Flash Cards
Why chunk files for sync? — To upload/download only changed parts and deduplicate identical content across files.
Metadata service vs block storage? — Metadata tracks chunk-to-file/version mappings; block storage holds the actual chunk bytes.
How are conflicts detected? — Via version vectors or last-known-version checks when two devices edit offline.
Typical chunk size? — Often around 4MB, balancing deduplication granularity against metadata overhead.