Blocks and Replication
HDFS never stores a file as one contiguous unit the way a local filesystem might; instead, it splits every file into fixed-size blocks, 128 MB by default in Hadoop 3.x, and distributes those blocks across different DataNodes. A 300 MB file, for instance, becomes three blocks: two full 128 MB blocks and one 44 MB block, each stored and replicated independently, which is why HDFS handles enormous files efficiently but wastes little space on the final partial block since blocks only consume the disk space they actually use.
Cricket analogy: Like a long Test innings broken into overs bowled from alternating ends, a 300 MB file is split into blocks handled by different DataNodes, each chunk manageable on its own rather than one unbroken 90-over spell.
Why Replication Exists
Commodity hardware fails routinely at scale: disks die, network links flap, and entire nodes go offline. HDFS compensates by replicating every block, three copies by default, controlled by the dfs.replication property, so that losing any single DataNode never loses data. The NameNode continuously monitors the actual replica count for every block against the desired replication factor, and whenever a DataNode dies or a disk corrupts a block, it schedules a new copy from a healthy replica to bring the count back up.
Cricket analogy: Like a captain always having a backup wicketkeeper training on the sidelines because gloves get injured, HDFS keeps three copies of every block so a failed disk never means lost data.
# Check the replication factor of a specific file
hdfs fs -stat %r /user/data/sales.csv
# 3
# Change replication factor for a file to 2
hdfs dfs -setrep -w 2 /user/data/sales.csv
# Configure the cluster-wide default in hdfs-site.xml
# <property>
# <name>dfs.replication</name>
# <value>3</value>
# </property>Replica Placement Policy
The default HDFS placement policy is rack-aware: the first replica is placed on the DataNode writing the block (or a random node if the write is coming from outside the cluster), the second replica goes to a different rack, and the third replica goes to another node on that same second rack. This spreads risk across at least two racks while keeping two of the three replicas close together to limit inter-rack network traffic during writes, and it means a single rack failure can never take out every copy of a block.
Cricket analogy: Like a team management placing squad members across two different team hotels during a tour, so a single hotel's plumbing failure doesn't leave the entire squad stranded before a match.
Beyond the classic 3x replication scheme, Hadoop 3.x introduced Erasure Coding as an alternative for colder, less frequently accessed data. Erasure Coding uses parity blocks (similar to RAID) to achieve the same fault tolerance at roughly 1.5x storage overhead instead of 3x, at the cost of higher CPU usage during reconstruction and reduced write/read performance for hot data.
Checksums and Block Integrity
Replication protects against losing an entire node, but it doesn't automatically protect against silent bit-rot on a healthy disk. Every DataNode computes and stores a checksum for each block when it's written, and clients verify that checksum on every read; if a mismatch is detected, the client transparently retries the read from a different replica while the NameNode is notified to schedule a fresh, verified copy of that block from a healthy source.
Cricket analogy: Like an umpire checking the ball's seam condition every few overs even though a spare ball is always ready, HDFS checksums catch quiet corruption on a disk even though replicas already exist as backup.
Setting dfs.replication to 1 removes all fault tolerance for that data: if the single DataNode holding the block fails, the data is permanently lost. Replication factor 1 should only be used for genuinely disposable intermediate data, never for anything the business depends on.
- HDFS splits files into fixed-size blocks (128 MB default in Hadoop 3.x) stored across DataNodes.
- Each block is replicated (3x by default) so losing any single DataNode never loses data.
- The default placement policy spreads replicas across at least two racks for fault tolerance.
- The NameNode continuously monitors and repairs under-replicated blocks automatically.
- Erasure Coding offers a lower-storage-overhead alternative to 3x replication for cold data.
- Checksums detect silent block corruption independently of replication.
- Setting replication factor to 1 eliminates fault tolerance and should be used only for disposable data.
Practice what you learned
1. What is the default HDFS block size and default replication factor in Hadoop 3.x?
2. Where does the default rack-aware placement policy put the second and third replicas of a block?
3. What triggers the NameNode to schedule re-replication of a block?
4. What does HDFS Erasure Coding trade off compared to classic 3x replication?
5. What is the risk of setting dfs.replication to 1 for important data?
Was this page helpful?
You May Also Like
HDFS Architecture
How HDFS organizes a cluster around a master/worker design so it can store and stream massive files reliably across commodity hardware.
NameNode and DataNodes
A closer look at HDFS's two core daemons: the NameNode that governs metadata and the DataNodes that physically store block data.
HDFS Read/Write Workflow
Step-by-step walkthrough of what actually happens inside HDFS when a client writes a new file or reads an existing one.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics