HDFS Architecture and Data Locality — Big Data & Distributed Computing

The Hadoop Distributed File System (HDFS) stores files from gigabytes to petabytes reliably across commodity machines expected to fail regularly. HDFS achieves this by splitting each file into 128MB blocks, distributing those blocks across DataNodes, and replicating each block three times across different nodes and racks. The architecture is optimised for write-once, read-many sequential access, where large files are scanned rather than randomly accessed — the dominant access pattern for batch analytics workloads.

Data locality is HDFS's key performance principle: move computation to the data rather than the data to the computation. When a MapReduce job needs a block, the framework schedules the mapper on a node that already holds it locally, eliminating network transfer entirely. In a cluster where network bandwidth is a shared resource, processing 128MB at local disk speed is orders of magnitude more efficient than transferring the block across the network first. This is why HDFS and MapReduce were designed as a co-located system rather than independent services.

Analogy🏏Cricket

🏏 Think of it like cricket: Imagine the DRS review system deployed across three independent video-review centres in Mumbai, Chennai, and London, each holding a copy of the ball-tracking data. A CAP partition is a network outage that cuts communication between them. A CP system says: if the centres cannot synchronise, no review decision is issued — no player walks until the system is restored. Consistency is guaranteed; availability is sacrificed. An AP system says: each centre issues its own decision based on its local data, even if that means Mumbai says out and London says not out — reviews continue but different centres may give different verdicts. Partition tolerance is non-negotiable because the network always has the possibility of failing; the choice is whether umpires wait for consensus or proceed with local data.