Setting Up a Hadoop Cluster

Analogy categoryAnalogies

Setting up Hadoop can mean three very different things depending on scale: standalone mode for local development with no daemons running, pseudo-distributed mode where every daemon runs on a single machine as separate JVM processes (ideal for learning and testing), or fully distributed mode across many physical or virtual nodes for production, each requiring different configuration files under $HADOOP_HOME/etc/hadoop.

🏏

Cricket analogy: Standalone mode is like a solo net practice session, pseudo-distributed is like an intra-squad practice match with one team playing both sides, and fully distributed is like an actual international match with two full teams across a real stadium.

Key Configuration Files

core-site.xml sets cluster-wide properties like fs.defaultFS (the NameNode's address); hdfs-site.xml controls replication factor (dfs.replication) and NameNode/DataNode storage directories; yarn-site.xml configures the ResourceManager and NodeManager behavior including memory and vcore allocation per node; mapred-site.xml specifies which framework (yarn) MapReduce jobs should run on.

🏏

Cricket analogy: core-site.xml naming the NameNode address is like the team sheet naming who's captain; hdfs-site.xml's replication factor is like deciding how many reserve batsmen to carry; yarn-site.xml's resource allocation is like assigning overs per bowler.

Planning Node Roles

A production cluster separates master roles (NameNode, ResourceManager, and often a Secondary NameNode or standby NameNode for HA) from worker roles (DataNode plus NodeManager, co-located on each worker so compute happens near stored data), and typically dedicates a rack-aware topology script so Hadoop places replicas across racks, not just across nodes, to survive a rack-level failure.

🏏

Cricket analogy: Separating NameNode/ResourceManager (masters) from DataNode/NodeManager (workers) is like separating the captain and coach from the playing eleven; rack-aware replica placement is like spreading key batsmen across different net sessions so a single injury doesn't wipe out the top order.

xml

<!-- core-site.xml -->
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://namenode-host:9000</value>
  </property>
</configuration>

<!-- hdfs-site.xml -->
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>
</configuration>

bash

# Format the NameNode only ONCE, on first setup
$HADOOP_HOME/bin/hdfs namenode -format

# Start HDFS and YARN daemons
$HADOOP_HOME/sbin/start-dfs.sh
$HADOOP_HOME/sbin/start-yarn.sh

# Verify local daemons are running
jps

Common Pitfalls

Forgetting that the NameNode should be formatted only once (rerunning hdfs namenode -format on a live cluster wipes existing metadata), mismatched SSH passwordless-login setup between master and workers, and firewall rules blocking the default HDFS (9000/8020) and YARN (8088/8032) ports are the most frequent causes of a cluster that 'starts' but reports zero live DataNodes.

Hadoop can run in standalone, pseudo-distributed, or fully distributed mode depending on scale.
core-site.xml, hdfs-site.xml, yarn-site.xml, and mapred-site.xml each control a different layer of the cluster.
dfs.replication in hdfs-site.xml controls how many copies of each block are stored.
Production clusters separate master roles (NameNode, ResourceManager) from worker roles (DataNode, NodeManager).
Rack-aware topology places replicas across racks to survive rack-level failures, not just node failures.
Formatting the NameNode wipes existing metadata, so it should only be done once on initial setup.
Passwordless SSH and open HDFS/YARN ports are prerequisites often missed when a cluster fails to start correctly.

Practice what you learned

1. Which Hadoop deployment mode runs every daemon on a single machine as separate JVM processes?

2. Which configuration file sets the HDFS block replication factor?

3. What happens if you rerun hdfs namenode -format on a live production cluster?

4. Why does Hadoop use rack-aware replica placement?

5. Which configuration file specifies that MapReduce jobs should run on YARN?

Was this page helpful?

The Hadoop Ecosystem

Hadoop Daemons

Related Study Notes in Programming

Browse all study notes

Programming

Apache Spark Study Notes

Programming · 30 topics

Programming

Apache Flink Study Notes

Programming · 30 topics

Programming

Snowflake Study Notes

Programming · 30 topics

Programming

Apache Airflow Study Notes

Programming · 30 topics

Programming

dbt (Data Build Tool) Study Notes

Programming · 30 topics

Programming

PowerShell Study Notes

Programming · 30 topics

Topics covered

#Programming#HadoopStudyNotes#SettingUpAHadoopCluster#Setting#Hadoop#Cluster#Key#StudyNotes#SkillVeris#ExamPrep

Setting Up a Hadoop Cluster