Blocks and Replication

HDFS never stores a file as one contiguous unit the way a local filesystem might; instead, it splits every file into fixed-size blocks, 128 MB by default in Hadoop 3.x, and distributes those blocks across different DataNodes. A 300 MB file, for instance, becomes three blocks: two full 128 MB blocks and one 44 MB block, each stored and replicated independently, which is why HDFS handles enormous files efficiently but wastes little space on the final partial block since blocks only consume the disk space they actually use.

Commodity hardware fails routinely at scale: disks die, network links flap, and entire nodes go offline. HDFS compensates by replicating every block, three copies by default, controlled by the dfs.replication property, so that losing any single DataNode never loses data. The NameNode continuously monitors the actual replica count for every block against the desired replication factor, and whenever a DataNode dies or a disk corrupts a block, it schedules a new copy from a healthy replica to bring the count back up.

The default HDFS placement policy is rack-aware: the first replica is placed on the DataNode writing the block (or a random node if the write is coming from outside the cluster), the second replica goes to a different rack, and the third replica goes to another node on that same second rack. This spreads risk across at least two racks while keeping two of the three replicas close together to limit inter-rack network traffic during writes, and it means a single rack failure can never take out every copy of a block.

Replication protects against losing an entire node, but it doesn't automatically protect against silent bit-rot on a healthy disk. Every DataNode computes and stores a checksum for each block when it's written, and clients verify that checksum on every read; if a mismatch is detected, the client transparently retries the read from a different replica while the NameNode is notified to schedule a fresh, verified copy of that block from a healthy source.