NameNode and DataNodes
Every HDFS cluster runs two kinds of daemons. The NameNode is the single authoritative process that owns the filesystem namespace: it knows every directory, every file, and which blocks make up each file, along with which DataNodes currently hold copies of those blocks. DataNodes are the workhorses; each one manages the blocks stored on its local disks, serves read and write traffic directly to clients, and reports in regularly so the NameNode's in-memory view stays accurate.
Cricket analogy: Like a franchise's team manager who knows every player's contract and squad number but never walks onto the field, while the players themselves, the DataNodes, physically deliver every run and wicket.
What the NameNode Tracks
The NameNode's metadata is deliberately small relative to the data it describes: it stores the namespace tree and file-to-block mapping durably on disk as the FsImage plus a running EditLog of changes, but it never persists the block-to-DataNode mapping to disk. That mapping is rebuilt in memory every time the NameNode starts, from the initial block reports every DataNode sends during startup, which is why a fresh cluster needs a brief period before it exits safe mode.
Cricket analogy: Like a scorer who keeps the official scorebook (FsImage) permanently but re-confirms the current fielding positions (block locations) fresh at the start of every session rather than writing them into the permanent record.
# Inspect NameNode metadata files on disk
ls -la $HADOOP_HOME/dfs/name/current
# fsimage_0000000000000012345
# edits_0000000000000012346-0000000000000012400
# VERSION
# View a human-readable dump of the current FsImage
hdfs oiv -i fsimage_0000000000000012345 -o /tmp/fsimage.txt -p DelimitedDataNode Responsibilities
Each DataNode stores its assigned blocks as ordinary files on its local disk, computed a checksum for every block to detect corruption, and exposes both a data-transfer port for streaming block reads and writes and an RPC channel for control messages from the NameNode. On startup, a DataNode sends a full block report listing every block it holds; afterward it sends lightweight heartbeats roughly every three seconds and incremental block reports whenever blocks are added or removed, which the NameNode uses to detect dead nodes and trigger re-replication.
Cricket analogy: Like a fielder who checks the ball's condition (checksum) before every throw and radios in (heartbeat) every few overs to confirm they're still fit and positioned, prompting a substitution if they go quiet.
If the NameNode doesn't receive a heartbeat from a DataNode for a configurable timeout (default 10.5 minutes, derived from heartbeat.recheck.interval and dfs.heartbeat.interval), it marks that DataNode as dead, removes it from the live-node list, and schedules re-replication of every block that node held so the replication factor is restored elsewhere in the cluster.
Secondary NameNode vs. High Availability
A common misconception is that the Secondary NameNode is a hot standby; it is not. Its only job in a classic (non-HA) deployment is periodic checkpointing: it fetches the current FsImage and EditLog from the NameNode, merges them into a new FsImage, and ships it back, preventing the EditLog from growing unbounded. Modern production clusters instead run HDFS High Availability, where a Standby NameNode continuously replays the same EditLog stream (via JournalNodes) and can be promoted to Active within seconds if the primary fails.
Cricket analogy: Like a team's video analyst who compiles a fresh highlights reel periodically (checkpointing) but never actually walks in to bat, unlike a genuine impact substitute (Standby NameNode) who is match-fit and ready to take the crease instantly if the opener retires hurt.
Do not treat the Secondary NameNode as a failover target: it holds a periodically merged checkpoint, not a live replica, so promoting it manually after a NameNode crash can lose any edits made since the last checkpoint. For real failover guarantees, configure HDFS HA with Active/Standby NameNodes backed by JournalNodes or a shared NFS edits directory.
- The NameNode owns the namespace and metadata; DataNodes own the physical block storage.
- The block-to-DataNode mapping lives only in memory and is rebuilt from DataNode block reports on startup.
- The FsImage and EditLog persist namespace metadata durably on the NameNode's disk.
- DataNodes send heartbeats every few seconds and block reports to keep the NameNode's view current.
- A missed heartbeat past the timeout marks a DataNode dead and triggers re-replication of its blocks.
- The Secondary NameNode only performs periodic checkpointing; it is not a hot standby.
- HDFS High Availability uses an Active/Standby NameNode pair with JournalNodes for real failover.
Practice what you learned
1. Where does the NameNode's block-to-DataNode location mapping live?
2. What triggers the NameNode to mark a DataNode as dead?
3. What is the actual role of the Secondary NameNode in a non-HA deployment?
4. What provides real automatic failover for the NameNode in production clusters?
5. What does a DataNode send to the NameNode immediately upon startup?
Was this page helpful?
You May Also Like
HDFS Architecture
How HDFS organizes a cluster around a master/worker design so it can store and stream massive files reliably across commodity hardware.
Blocks and Replication
How HDFS splits files into large blocks and replicates them across the cluster to guarantee durability and availability.
HDFS Read/Write Workflow
Step-by-step walkthrough of what actually happens inside HDFS when a client writes a new file or reads an existing one.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics