HDFS Architecture

HDFS (Hadoop Distributed File System) is the primary storage layer of the Hadoop ecosystem, built to hold very large datasets, often terabytes to petabytes, across a cluster of inexpensive commodity servers. It follows a master/worker architecture: a single NameNode holds the filesystem namespace and metadata, while many DataNodes store the actual data as fixed-size blocks. Clients talk to the NameNode to locate data, then stream blocks directly to and from DataNodes, keeping metadata operations separate from bulk data transfer.

The NameNode maintains the directory tree, file-to-block mapping, and block-to-DataNode location table entirely in memory for fast lookups, persisting only the namespace structure to disk via the FsImage and EditLog. DataNodes, by contrast, are simple workers: they store blocks on local disk, serve read and write requests from clients, and periodically send heartbeats and block reports back to the NameNode so it knows the cluster's real-time state.

HDFS was designed around a write-once-read-many access pattern optimized for high-throughput streaming reads of large files rather than low-latency random access, which is why it favors large block sizes and sequential I/O over the small, random reads typical of traditional filesystems. This trade-off means HDFS is a poor fit for storing millions of tiny files or serving interactive, low-latency queries, but it excels at batch analytics over massive datasets.

Because DataNodes are stateless workers that only need to register with the NameNode, HDFS scales horizontally simply by adding more commodity machines to the cluster without downtime. The rack-awareness feature lets the NameNode learn which rack each DataNode belongs to, so it can place block replicas across different racks to survive a rack-level failure while still minimizing cross-rack network traffic for reads.