HDFS Commands

HDFS exposes a POSIX-like command-line interface through the hdfs dfs (or the older hadoop fs) command, which mirrors familiar Unix shell verbs such as ls, mkdir, cp, and rm but operates against the distributed filesystem instead of local disk. Every command is prefixed with hdfs dfs followed by a dash and the verb, and paths default to the HDFS namespace unless explicitly written as a local file:// URI, which is the single most common source of confusion for people new to the tool.

The most frequently used commands cover the full lifecycle of moving data in and out of the cluster: -mkdir creates directories, -put and -copyFromLocal upload data from the local filesystem into HDFS, -get and -copyToLocal download data back out, -ls lists directory contents, -cat streams a file's contents to stdout, and -rm deletes files (requiring -r for directories and -skipTrash to bypass the trash folder entirely). Because HDFS files are typically enormous, -cat is usually piped through head or tail rather than dumped raw to a terminal.

Beyond basic file movement, hdfs dfs offers commands for capacity planning and access control that mirror standard Unix tooling: -du reports disk usage for a path (with -s to summarize and -h for human-readable units), -chmod and -chown adjust POSIX-style permissions and ownership, and -setrep changes a file's replication factor on the fly. The -df -h command instead reports overall cluster capacity, which is useful for spotting when a cluster is approaching full before jobs start failing on write.

New users frequently forget that -rm without -skipTrash moves files into a per-user trash directory rather than deleting them immediately, which is safer but silently continues to consume disk quota until the trash interval expires; conversely, using -skipTrash by habit on important data risks unrecoverable deletion. Another common mistake is running hdfs dfs -put with a relative local path from the wrong working directory, or forgetting -p on -mkdir when parent directories don't yet exist, both of which produce confusing 'No such file or directory' errors.