NameNode and DataNodes

Every HDFS cluster runs two kinds of daemons. The NameNode is the single authoritative process that owns the filesystem namespace: it knows every directory, every file, and which blocks make up each file, along with which DataNodes currently hold copies of those blocks. DataNodes are the workhorses; each one manages the blocks stored on its local disks, serves read and write traffic directly to clients, and reports in regularly so the NameNode's in-memory view stays accurate.

The NameNode's metadata is deliberately small relative to the data it describes: it stores the namespace tree and file-to-block mapping durably on disk as the FsImage plus a running EditLog of changes, but it never persists the block-to-DataNode mapping to disk. That mapping is rebuilt in memory every time the NameNode starts, from the initial block reports every DataNode sends during startup, which is why a fresh cluster needs a brief period before it exits safe mode.

Each DataNode stores its assigned blocks as ordinary files on its local disk, computed a checksum for every block to detect corruption, and exposes both a data-transfer port for streaming block reads and writes and an RPC channel for control messages from the NameNode. On startup, a DataNode sends a full block report listing every block it holds; afterward it sends lightweight heartbeats roughly every three seconds and incremental block reports whenever blocks are added or removed, which the NameNode uses to detect dead nodes and trigger re-replication.

A common misconception is that the Secondary NameNode is a hot standby; it is not. Its only job in a classic (non-HA) deployment is periodic checkpointing: it fetches the current FsImage and EditLog from the NameNode, merges them into a new FsImage, and ships it back, preventing the EditLog from growing unbounded. Modern production clusters instead run HDFS High Availability, where a Standby NameNode continuously replays the same EditLog stream (via JournalNodes) and can be promoted to Active within seconds if the primary fails.