Quick Reference Overview
This reference collects the HDFS shell commands, YARN/job-tracking commands, and configuration properties used most often in day-to-day Hadoop work, so you don't need to dig through full documentation for routine tasks like checking cluster health, submitting a job, or looking up a default port. It assumes you already understand the concepts covered elsewhere in this course and is meant purely as a fast lookup.
Cricket analogy: Is like a fielding captain's laminated card of standard field placements for different bowlers — not a coaching manual, just the fast lookup you glance at between overs.
Common HDFS Commands
The hdfs dfs (or legacy hadoop fs) command family mirrors familiar Unix shell verbs — ls, mkdir, rm, cat — but operates against the distributed filesystem, and every path can be prefixed with hdfs://namenode:port/ or left relative to the user's HDFS home directory. Beyond basic file operations, hdfs dfsadmin -report gives a cluster-wide capacity and DataNode health summary, and hdfs fsck validates block health and replication, both of which are the first commands to run when diagnosing 'why is my job slow or failing.'
Cricket analogy: Is like a scorer's shorthand notation system — a quick 'W' for wicket or '4' for boundary mirrors familiar cricket scoring conventions, just applied consistently across every match (path) in the tournament archive.
# File operations (mirror Unix shell verbs)
hdfs dfs -ls /user/data
hdfs dfs -mkdir -p /user/data/staging
hdfs dfs -put localfile.csv /user/data/staging/
hdfs dfs -get /user/data/output/part-r-00000 ./results.txt
hdfs dfs -cat /user/data/staging/localfile.csv | head
hdfs dfs -rm -r /user/data/staging
# Cluster diagnostics
hdfs dfsadmin -report # capacity + DataNode health summary
hdfs fsck /user/data -files -blocks # block health and replication check
hdfs dfs -du -h /user/data # human-readable directory sizesMapReduce and YARN Cheat Sheet
Jobs are submitted with hadoop jar <jarfile> <MainClass> <args>, and once running, yarn application -list shows all running applications with their IDs, while yarn logs -applicationId <id> pulls aggregated container logs after completion — essential for debugging a failed task without SSHing into individual nodes. yarn node -list shows NodeManager health across the cluster, and killing a runaway job cleanly uses yarn application -kill <applicationId> rather than manually terminating processes, which leaves YARN's resource accounting in an inconsistent state.
Cricket analogy: Is like a match official's standard toolkit — the toss coin (job submission), the live scoreboard feed (application -list), and the post-match report (logs) are all reached for through the same standard procedures every match.
# Submit and monitor a job
hadoop jar wordcount.jar WordCountDriver /input /output
yarn application -list # all running apps
yarn application -status <applicationId> # detail on one app
yarn logs -applicationId <applicationId> # aggregated container logs
yarn application -kill <applicationId> # cleanly terminate a job
yarn node -list -all # NodeManager health across clusterKey Configuration Properties and Defaults
A handful of properties across core-site.xml, hdfs-site.xml, and yarn-site.xml account for most day-to-day tuning: dfs.replication (default 3, replication factor per block), dfs.blocksize (default 128MB in Hadoop 2.x/3.x), yarn.nodemanager.resource.memory-mb (total memory a NodeManager can allocate to containers), and yarn.scheduler.minimum-allocation-mb (smallest container size the scheduler will grant). Default service ports worth memorizing include the NameNode web UI at 9870 (Hadoop 3.x; 50070 in Hadoop 2.x), the ResourceManager web UI at 8088, and the HDFS RPC port at 8020 (or 9000 in some 2.x configurations).
Cricket analogy: Is like knowing the standard dimensions baked into every cricket ground — a 22-yard pitch, boundary roughly 65-90 yards — defaults you only override for a special exhibition match.
Quick lookup: dfs.replication=3, dfs.blocksize=128MB, yarn.nodemanager.resource.memory-mb (cluster-dependent, no universal default), NameNode UI=9870 (Hadoop 3.x), ResourceManager UI=8088, HDFS RPC=8020.
Port numbers changed between Hadoop 2.x and 3.x — most notably the NameNode web UI moved from 50070 to 9870. Always confirm the version before trusting a memorized port number, especially in mixed-version documentation or older tutorials.
hdfs dfscommands mirror Unix shell verbs (ls, mkdir, rm, cat, put, get) against the distributed filesystem.hdfs dfsadmin -reportandhdfs fsckare the first diagnostic commands for cluster health and block replication issues.- Submit jobs with
hadoop jar <jar> <MainClass> <args>; monitor withyarn application -listandyarn logs -applicationId. - Always kill runaway jobs with
yarn application -kill <id>rather than manually terminating processes. - Default replication factor is 3, default block size is 128MB in Hadoop 2.x/3.x.
- NameNode web UI is on port 9870 in Hadoop 3.x (50070 in 2.x); ResourceManager web UI is on 8088; HDFS RPC is on 8020.
- Confirm the Hadoop major version before trusting memorized port numbers, since key ports changed between 2.x and 3.x.
Practice what you learned
1. Which command gives a cluster-wide capacity and DataNode health summary?
2. What is the default HDFS block size in Hadoop 2.x/3.x?
3. What is the correct way to terminate a runaway YARN application?
4. Which port hosts the NameNode web UI by default in Hadoop 3.x?
5. Which command pulls aggregated container logs for a completed YARN application?
Was this page helpful?
You May Also Like
Hadoop Best Practices
A practical guide to designing, tuning, and operating Hadoop clusters so they stay reliable and performant as data volumes grow.
Building a Word Count Job
A hands-on walkthrough of writing, compiling, and running the canonical Hadoop MapReduce word count program from mapper to driver.
Hadoop Interview Questions
The core HDFS, MapReduce, and YARN questions Hadoop interviewers ask most often, with the reasoning behind strong answers.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics