HDFS Commands
HDFS exposes a POSIX-like command-line interface through the hdfs dfs (or the older hadoop fs) command, which mirrors familiar Unix shell verbs such as ls, mkdir, cp, and rm but operates against the distributed filesystem instead of local disk. Every command is prefixed with hdfs dfs followed by a dash and the verb, and paths default to the HDFS namespace unless explicitly written as a local file:// URI, which is the single most common source of confusion for people new to the tool.
Cricket analogy: Like a coach using the same hand signals across every ground worldwide, hdfs dfs uses familiar Unix-style verbs so a shell user already knows most of the vocabulary instantly.
Basic File Operations
The most frequently used commands cover the full lifecycle of moving data in and out of the cluster: -mkdir creates directories, -put and -copyFromLocal upload data from the local filesystem into HDFS, -get and -copyToLocal download data back out, -ls lists directory contents, -cat streams a file's contents to stdout, and -rm deletes files (requiring -r for directories and -skipTrash to bypass the trash folder entirely). Because HDFS files are typically enormous, -cat is usually piped through head or tail rather than dumped raw to a terminal.
Cricket analogy: Like a groundskeeper's daily checklist covering pitch setup, rolling, and covering, hdfs dfs's core verbs cover the full lifecycle of getting data in, viewing it, and clearing it out.
# Create a directory
hdfs dfs -mkdir -p /user/analytics/raw
# Upload a local file into HDFS
hdfs dfs -put sales_2026.csv /user/analytics/raw/
# List directory contents with human-readable sizes
hdfs dfs -ls -h /user/analytics/raw
# View the first lines of a large file without dumping it all
hdfs dfs -cat /user/analytics/raw/sales_2026.csv | head -n 20
# Download a file back to local disk
hdfs dfs -get /user/analytics/raw/sales_2026.csv ./local_copy.csv
# Remove a directory recursively, skipping trash
hdfs dfs -rm -r -skipTrash /user/analytics/tmpManaging Space, Permissions, and Replication
Beyond basic file movement, hdfs dfs offers commands for capacity planning and access control that mirror standard Unix tooling: -du reports disk usage for a path (with -s to summarize and -h for human-readable units), -chmod and -chown adjust POSIX-style permissions and ownership, and -setrep changes a file's replication factor on the fly. The -df -h command instead reports overall cluster capacity, which is useful for spotting when a cluster is approaching full before jobs start failing on write.
Cricket analogy: Like a groundsman checking pitch moisture readings before a Test, hdfs dfs -du checks storage usage before a big data load, catching space problems before they cause a job failure mid-match.
hdfs dfs commands operate on HDFS for regular users, while cluster-wide administrative tasks (safe mode, decommissioning nodes, generating the cluster health report) use the separate hdfs dfsadmin command, which typically requires superuser privileges: hdfs dfsadmin -report, hdfs dfsadmin -safemode get, and hdfs dfsadmin -refreshNodes are common examples.
Common Pitfalls
New users frequently forget that -rm without -skipTrash moves files into a per-user trash directory rather than deleting them immediately, which is safer but silently continues to consume disk quota until the trash interval expires; conversely, using -skipTrash by habit on important data risks unrecoverable deletion. Another common mistake is running hdfs dfs -put with a relative local path from the wrong working directory, or forgetting -p on -mkdir when parent directories don't yet exist, both of which produce confusing 'No such file or directory' errors.
Cricket analogy: Like a fielder assuming a caught ball is automatically given out without checking with the third umpire, forgetting that -rm moves files to trash rather than deleting instantly can cause confusion about whether space was actually freed.
hdfs dfs -rm -r -skipTrash permanently deletes data immediately, bypassing the trash directory entirely. There is no undo. Reserve -skipTrash for genuinely disposable scratch data, and always double-check the path before running it against anything in a shared cluster.
- hdfs dfs mirrors familiar Unix verbs (ls, mkdir, cp, rm) but operates against the distributed filesystem.
- -put/-copyFromLocal upload data; -get/-copyToLocal download it; -cat streams file contents.
- -du reports per-path disk usage; -df -h reports overall cluster capacity.
- -chmod and -chown manage POSIX-style permissions and ownership on HDFS paths.
- -setrep changes a file's replication factor without re-uploading it.
- Administrative tasks (safe mode, node reports) use the separate hdfs dfsadmin command.
- -rm without -skipTrash moves files to trash; -skipTrash deletes permanently and immediately.
Practice what you learned
1. Which command uploads a local file into HDFS?
2. What does hdfs dfs -du -h report?
3. What happens by default when you run hdfs dfs -rm on a file (without -skipTrash)?
4. Which command is used for cluster-wide administrative tasks like checking safe mode status?
5. Which command changes a file's replication factor after it has already been written?
Was this page helpful?
You May Also Like
HDFS Architecture
How HDFS organizes a cluster around a master/worker design so it can store and stream massive files reliably across commodity hardware.
HDFS Read/Write Workflow
Step-by-step walkthrough of what actually happens inside HDFS when a client writes a new file or reads an existing one.
Blocks and Replication
How HDFS splits files into large blocks and replicates them across the cluster to guarantee durability and availability.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics