How to Design a Distributed File System
Learn how to design a distributed file system: chunking, metadata service, replication, rack-aware placement, and failure recovery.
Expected Interview Answer
A distributed file system splits large files into fixed-size chunks replicated across many storage nodes, with a separate metadata service tracking which chunks make up each file and where their replicas live, so the system scales storage and throughput horizontally while tolerating node failures.
A master or metadata service (like GFS’s master or HDFS’s NameNode) stores the file-to-chunk mapping and chunk-to-node locations in memory for fast lookups, while the actual chunk data — typically 64-128MB blocks — lives on many chunk-server nodes, each chunk replicated three or more times across different racks for durability. Clients ask the metadata service which nodes hold the chunks they need, then read or write chunk data directly from those chunk servers, keeping the metadata service off the data path so it does not bottleneck throughput. Writes are coordinated via a primary replica that orders mutations and forwards them to secondaries, and periodic heartbeats let the metadata service detect dead nodes and trigger re-replication to maintain the target replication factor. This master-plus-chunkserver split, combined with chunk-level checksums and rack-aware replica placement, is what lets systems like HDFS and GFS reliably store petabytes across thousands of commodity machines.
- Splits huge files into chunks so storage and throughput scale horizontally across nodes
- Replication across nodes and racks tolerates individual machine or rack failures
- Keeping metadata separate from data keeps the metadata service from bottlenecking reads/writes
- Automatic re-replication on node failure keeps durability guarantees intact without manual intervention
AI Mentor Explanation
A distributed file system is like splitting a full match’s footage into over-by-over reels, with a production office keeping a master index of which storage vault holds each reel, rather than storing the whole match as one giant tape in one place. Each reel is duplicated across three separate vaults in case one building floods or loses power. When a broadcaster wants over 34, they ask the index office which vault has it and fetch the reel directly, never routing the actual footage through the office itself. If a vault goes offline, the index office notices via regular check-ins and orders a fresh copy made elsewhere to keep three copies alive.
Step-by-Step Explanation
Step 1
Split files into chunks
Large files are divided into fixed-size chunks (e.g. 64-128MB) that can be stored independently across nodes.
Step 2
Track metadata centrally
A metadata service stores the file-to-chunk mapping and chunk-to-node locations, kept off the actual data path.
Step 3
Replicate chunks across nodes and racks
Each chunk is stored on 3+ chunk servers across different racks for durability against node or rack failure.
Step 4
Detect failures and re-replicate
Heartbeats let the metadata service detect dead nodes and trigger new replicas to restore the target replication factor.
What Interviewer Expects
- Separates metadata (file-to-chunk mapping) from actual chunk data storage
- Explains chunking with a concrete size and a replication factor (typically 3)
- Mentions rack-aware placement and failure detection via heartbeats
- Names a real system (HDFS, GFS, Ceph) and how clients bypass the metadata service for actual I/O
Common Mistakes
- Routing all file data through the metadata/master service, making it a throughput bottleneck
- Forgetting replication entirely or using a single copy per chunk
- Ignoring failure detection and automatic re-replication
- Treating the metadata service as a single point of failure with no standby/backup
Best Answer (HR Friendly)
“A distributed file system breaks large files into chunks and spreads copies of each chunk across many machines, so no single disk failure ever loses data and the system can handle huge files and lots of traffic. A lightweight index keeps track of which machine has which chunk, so clients can go straight to the right machine instead of funneling everything through one bottleneck.”
Code Example
CHUNK_SIZE = 64 * 1024 * 1024 # 64MB
REPLICATION_FACTOR = 3
class MetadataService:
def __init__(self):
self.file_to_chunks = {} # filename -> [chunk_id, ...]
self.chunk_to_nodes = {} # chunk_id -> [node_id, ...]
self.node_last_heartbeat = {} # node_id -> timestamp
def get_chunk_locations(self, filename):
chunk_ids = self.file_to_chunks[filename]
return {cid: self.chunk_to_nodes[cid] for cid in chunk_ids}
def record_heartbeat(self, node_id, now):
self.node_last_heartbeat[node_id] = now
def check_for_dead_nodes(self, now, timeout=30):
dead = [n for n, t in self.node_last_heartbeat.items() if now - t > timeout]
for node_id in dead:
self._re_replicate_chunks_on(node_id)
def _re_replicate_chunks_on(self, dead_node_id):
for chunk_id, nodes in self.chunk_to_nodes.items():
if dead_node_id in nodes:
nodes.remove(dead_node_id)
if len(nodes) < REPLICATION_FACTOR:
new_node = pick_healthy_node(exclude=nodes)
schedule_copy(chunk_id, source=nodes[0], target=new_node)
nodes.append(new_node)Follow-up Questions
- How does the metadata service avoid becoming a single point of failure?
- How do writes get coordinated across replicas (primary/secondary chunk servers)?
- How would you handle very small files that are much smaller than the chunk size?
- How does rack-aware placement improve durability compared to random placement?
MCQ Practice
1. Why does a distributed file system keep the metadata service off the actual data read/write path?
Clients fetch locations from metadata once, then read/write chunk data directly from chunk servers, keeping metadata lookups cheap and off the high-throughput data path.
2. What is a typical purpose of rack-aware replica placement?
Placing replicas across different racks (and often data centers) protects against correlated failures like a rack losing power.
3. What triggers re-replication of a chunk in a distributed file system?
Heartbeat monitoring lets the metadata service detect failed nodes and automatically schedule new replicas to restore the target replication factor.
Flash Cards
Why split files into chunks? — So storage and I/O throughput can scale horizontally across many nodes instead of one machine holding an entire large file.
What does the metadata service store? — The file-to-chunk mapping and which nodes hold each chunk — kept off the actual data transfer path.
Typical replication factor? — Usually 3, with replicas spread across different racks or data centers for durability.
How are dead nodes detected? — Via periodic heartbeats; a missed heartbeat triggers automatic re-replication of that node’s chunks.