How Would You Design a Blob Storage System?
Learn how blob storage systems chunk, replicate, and track large objects via a metadata manifest, with durability and scaling trade-offs.
Expected Interview Answer
A blob storage system stores unstructured binary objects (files, images, videos) by splitting each object into fixed-size chunks distributed across many storage nodes, tracked through a separate metadata layer that maps object keys to chunk locations, replicas, and versions.
The design splits into two planes: a metadata service (often a strongly consistent key-value or relational store) that records which chunks make up an object, their checksums, and where replicas live, and a data plane of storage nodes that hold the actual chunk bytes on disk. Writes stream the object into chunks, write each chunk to multiple nodes (or erasure-code it across nodes) for durability, then commit the chunk manifest to metadata only after enough replicas acknowledge. Reads look up the manifest, fetch chunks in parallel from the nearest healthy replicas, and reassemble them, often behind a CDN for hot objects. Durability comes from replication or erasure coding plus background scrubbing that detects and repairs corrupted or lost chunks, while availability comes from spreading replicas across failure domains (racks, availability zones).
- Separating metadata from bulk data lets each scale independently on its own hardware profile
- Chunking large objects enables parallel upload/download and partial-range reads
- Replication or erasure coding across failure domains protects against disk and node loss
- A CDN layer in front absorbs read traffic for hot objects without touching storage nodes
AI Mentor Explanation
Blob storage is like a stadium’s archive splitting a full match broadcast into individual over-by-over reels instead of storing one giant unbroken tape. A separate index card catalog (metadata) records exactly which shelf holds each over’s reel and keeps duplicate copies in two different storage rooms in case one floods. When someone requests the full match, the archivist pulls every reel in parallel from whichever room is closest and splices them back together. That split-store-track-reassemble pattern is exactly how a blob storage system handles a large file.
Step-by-Step Explanation
Step 1
Chunk the object on upload
The client or gateway splits the object into fixed-size chunks and computes a checksum for each.
Step 2
Write chunks to storage nodes
Each chunk is replicated (or erasure-coded) across nodes in different failure domains for durability.
Step 3
Commit the manifest
Once enough replicas acknowledge, the metadata service records the object key, chunk list, checksums, and locations.
Step 4
Serve reads via the manifest
A read looks up the manifest, fetches chunks in parallel from the nearest healthy replicas, and reassembles the object.
What Interviewer Expects
- Separates the metadata plane (object -> chunk manifest) from the data plane (chunk bytes on nodes)
- Explains durability via replication or erasure coding across failure domains
- Mentions chunking for parallel transfer and partial-range reads
- Discusses background scrubbing/repair and CDN caching for hot reads
Common Mistakes
- Treating blob storage as just “a big file on one disk” with no chunking
- Forgetting to separate metadata lookups from bulk data transfer
- Ignoring failure domains, so all replicas can be lost in one rack outage
- Not mentioning checksums or scrubbing, so silent corruption goes undetected
Best Answer (HR Friendly)
“Blob storage breaks large files into smaller chunks and spreads copies of those chunks across many machines, while a separate index keeps track of exactly where every chunk lives. That way uploads and downloads can happen in parallel, and if one machine fails there are backup copies elsewhere so nothing is lost.”
Code Example
CHUNK_SIZE = 4 * 1024 * 1024 # 4 MB
def upload_object(object_key, data_stream, storage_nodes, replicas=3):
manifest = {"key": object_key, "chunks": []}
index = 0
while True:
chunk = data_stream.read(CHUNK_SIZE)
if not chunk:
break
checksum = sha256(chunk)
target_nodes = pick_nodes(storage_nodes, replicas, chunk_id=index)
acked = 0
for node in target_nodes:
if node.write_chunk(object_key, index, chunk, checksum):
acked += 1
if acked < majority(replicas):
raise Exception("Not enough replicas acknowledged, aborting upload")
manifest["chunks"].append({
"index": index,
"checksum": checksum,
"nodes": [n.id for n in target_nodes],
})
index += 1
metadata_store.commit_manifest(object_key, manifest)
return manifestFollow-up Questions
- How would you handle a chunk that fails checksum validation on read?
- What is the difference between replication and erasure coding for durability, and when would you pick each?
- How would you support range reads (e.g., byte 1000-2000) without downloading the whole object?
- How would you design garbage collection for chunks whose objects have been deleted?
MCQ Practice
1. In a blob storage system, what is the primary role of the metadata service?
The metadata plane tracks which chunks form an object and where those chunks physically live, separate from the bulk data plane.
2. Why do blob storage systems chunk large objects instead of storing them as one contiguous file?
Chunking allows parallel upload/download, range reads, and per-chunk replication or repair without touching the whole object.
3. What technique lets a blob store recover from disk failures without storing full replicas of every chunk?
Erasure coding splits data into fragments with parity so the object can be reconstructed even if some fragments are lost, using less storage overhead than full replication.
Flash Cards
What are the two planes in a blob storage design? — A metadata plane (object-to-chunk manifest) and a data plane (chunk bytes on storage nodes).
Why chunk large objects? — To enable parallel transfer, partial-range reads, and independent per-chunk replication.
Erasure coding vs replication? — Erasure coding reconstructs data from parity fragments with less storage overhead; replication keeps full copies but is simpler to reason about.
How does a blob store detect silent corruption? — Per-chunk checksums verified on read, plus background scrubbing that repairs bad chunks from healthy replicas.