Hadoop Daemons
A Hadoop cluster is really a set of long-running Java processes (daemons) coordinating over the network. HDFS runs the NameNode (holds the filesystem namespace and block-location metadata entirely in memory) and one DataNode per worker (stores actual data blocks on local disk and reports block reports and heartbeats to the NameNode every few seconds).
Cricket analogy: The NameNode holding the whole filesystem's metadata in memory is like the scorer keeping the master scorebook that says who's where, while DataNodes are the fielders actually holding the ball at their positions, reporting back to the scorer via radio.
The NameNode and Secondary NameNode
The NameNode is a single point of failure in classic (non-HA) deployments — if it goes down, the cluster can't resolve any file-to-block mapping even though the data blocks still physically exist on DataNodes — so production clusters either run a Secondary NameNode (which periodically merges the edit log into the fsimage checkpoint; despite its misleading name it is NOT a hot standby) or configure full NameNode HA with a Standby NameNode and JournalNodes via the Quorum Journal Manager.
Cricket analogy: Losing the NameNode is like losing the only scorebook mid-match — even though every run was actually scored, nobody can reconstruct the innings; the Secondary NameNode photocopying the scorebook helps recovery but isn't a substitute umpire ready to take over instantly.
YARN's ResourceManager and NodeManager
YARN separates resource management from job scheduling: the ResourceManager (one per cluster, with an optional standby for HA) tracks total cluster capacity and grants containers to applications, while each NodeManager (one per worker) manages containers on its own node, monitoring their CPU and memory usage and killing any that exceed their allocation; per-job coordination is handled by an ApplicationMaster that itself runs inside a container.
Cricket analogy: The ResourceManager granting containers cluster-wide is like the tournament committee allocating practice nets and time slots to every team, while each NodeManager is like a ground's local groundskeeper enforcing that only the allotted slot is used at that specific venue.
$ jps
2145 NameNode
2231 DataNode
2350 ResourceManager
2401 NodeManager
2500 SecondaryNameNode
$ hdfs dfsadmin -report
Live datanodes (3):
Name: 10.0.0.11:9866 (worker1)
Hostname: worker1
Decommission Status : Normal
Configured Capacity: 500107862016 (465.76 GB)
DFS Used: 12582912 (12 MB)
DFS Remaining: 500095279104 (465.75 GB)Monitoring Daemon Health
Running jps on any node shows which daemons are alive locally (NameNode, DataNode, ResourceManager, NodeManager, SecondaryNameNode), while hdfs dfsadmin -report and the NameNode's web UI (typically port 9870 in Hadoop 3.x) give a cluster-wide view of live versus dead DataNodes, useful for spotting a silently crashed daemon before a job fails.
- A Hadoop cluster consists of long-running Java daemons coordinating over the network.
- The NameNode holds filesystem namespace and block-location metadata entirely in memory.
- DataNodes store actual data blocks on disk and send heartbeats/block reports to the NameNode.
- The classic NameNode is a single point of failure; the Secondary NameNode is a checkpointer, not a hot standby.
- NameNode HA uses a Standby NameNode plus JournalNodes via the Quorum Journal Manager.
- YARN's ResourceManager tracks cluster-wide capacity; each NodeManager enforces limits on its own node.
- jps and hdfs dfsadmin -report are the primary tools for checking whether daemons are alive.
Practice what you learned
1. Where does the NameNode keep the filesystem namespace and block-location metadata during normal operation?
2. What does the Secondary NameNode actually do?
3. What mechanism provides true NameNode high availability?
4. What does YARN's ResourceManager do?
5. Which command shows which Hadoop daemons are running locally on a machine?
Was this page helpful?
You May Also Like
What Is Hadoop?
An introduction to Apache Hadoop, the open-source framework that popularized distributed storage and processing of massive datasets on commodity hardware.
Setting Up a Hadoop Cluster
A practical walkthrough of planning and configuring a Hadoop cluster, from single-node pseudo-distributed setups to multi-node production layouts.
Your First Hadoop Job
A hands-on walkthrough of writing, packaging, and running a classic word count MapReduce job on Hadoop, from source to output.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics