Hadoop Quick Reference

This reference collects the HDFS shell commands, YARN/job-tracking commands, and configuration properties used most often in day-to-day Hadoop work, so you don't need to dig through full documentation for routine tasks like checking cluster health, submitting a job, or looking up a default port. It assumes you already understand the concepts covered elsewhere in this course and is meant purely as a fast lookup.

The hdfs dfs (or legacy hadoop fs) command family mirrors familiar Unix shell verbs — ls, mkdir, rm, cat — but operates against the distributed filesystem, and every path can be prefixed with hdfs://namenode:port/ or left relative to the user's HDFS home directory. Beyond basic file operations, hdfs dfsadmin -report gives a cluster-wide capacity and DataNode health summary, and hdfs fsck validates block health and replication, both of which are the first commands to run when diagnosing 'why is my job slow or failing.'

Jobs are submitted with hadoop jar <jarfile> <MainClass> <args>, and once running, yarn application -list shows all running applications with their IDs, while yarn logs -applicationId <id> pulls aggregated container logs after completion — essential for debugging a failed task without SSHing into individual nodes. yarn node -list shows NodeManager health across the cluster, and killing a runaway job cleanly uses yarn application -kill <applicationId> rather than manually terminating processes, which leaves YARN's resource accounting in an inconsistent state.

A handful of properties across core-site.xml, hdfs-site.xml, and yarn-site.xml account for most day-to-day tuning: dfs.replication (default 3, replication factor per block), dfs.blocksize (default 128MB in Hadoop 2.x/3.x), yarn.nodemanager.resource.memory-mb (total memory a NodeManager can allocate to containers), and yarn.scheduler.minimum-allocation-mb (smallest container size the scheduler will grant). Default service ports worth memorizing include the NameNode web UI at 9870 (Hadoop 3.x; 50070 in Hadoop 2.x), the ResourceManager web UI at 8088, and the HDFS RPC port at 8020 (or 9000 in some 2.x configurations).