HDFS Read/Write Workflow
Neither reads nor writes in HDFS ever move file data through the NameNode; it only ever hands out metadata and locations. When a client opens a file, it first asks the NameNode which blocks make up the file and which DataNodes hold each one, then it streams the actual bytes directly to or from those DataNodes over a separate data-transfer protocol. This split keeps the NameNode's workload proportional to the number of metadata operations rather than the volume of data flowing through the cluster, which is exactly why a single NameNode can coordinate a cluster moving many gigabytes per second.
Cricket analogy: Like a stadium's ticketing office that only tells you your seat number and gate, while you physically walk to the stand yourself, the NameNode only hands out block locations while data flows directly between client and DataNode.
The Write Path
When a client writes a new file, it first asks the NameNode to create a namespace entry, then requests a list of DataNodes to host the first block's replicas; the NameNode returns an ordered pipeline based on the replica placement policy. The client streams data in small packets (typically 64 KB chunks within a larger 512-byte-checksummed unit) to the first DataNode in that pipeline, which forwards each packet to the second, which forwards it to the third, and acknowledgments flow back up the same pipeline in reverse; only once every DataNode in the pipeline has acknowledged a packet does the client consider it durably written.
Cricket analogy: Like a fielding relay throw from the boundary to the keeper via a cutoff man, HDFS writes stream data through a pipeline of DataNodes, and only once the throw completes the full relay does the run-out (write) count as confirmed.
# Conceptual write sequence (simplified)
# 1. Client -> NameNode: create('/user/data/sales.csv')
# 2. NameNode -> Client: pipeline = [dn1, dn2, dn3]
# 3. Client -> dn1 -> dn2 -> dn3: stream packets
# 4. dn3 -> dn2 -> dn1 -> Client: acknowledgments flow back
# 5. Client -> NameNode: complete() once all blocks acked
hdfs dfs -put -f large_dataset.parquet /user/data/
# Client blocks until the final block's pipeline is fully acknowledgedThe Read Path
Reads follow a mirror-image but simpler flow: the client asks the NameNode for the block locations of a file, and the NameNode returns them sorted by proximity, same node first, then same rack, then off-rack, so the client naturally prefers the closest available replica. The client then opens a direct connection to the chosen DataNode and streams the block, verifying each chunk's checksum as it arrives; if a DataNode is unreachable or a checksum fails, the client transparently falls back to the next DataNode in the list without the application ever seeing an error.
Cricket analogy: Like a broadcaster always cutting to the nearest camera angle first for the fastest replay, HDFS reads prefer the closest DataNode replica, falling back to a farther one only if the nearest feed drops out.
The client library, DFSClient, is what performs all this pipeline and fallback logic transparently; application code calling the Hadoop FileSystem API (or hdfs dfs) never sees the individual DataNode connections, retries, or checksum verification, it just sees a standard read or write stream succeed or throw an IOException after all retries are exhausted.
Handling Failures Mid-Transfer
If a DataNode in a write pipeline fails mid-transfer, HDFS doesn't abort the write: it removes the failed node from the pipeline, asks the NameNode for a replacement DataNode, and continues streaming to the remaining nodes, with the block later re-replicated to restore the full replication factor once the write completes. This means a client's write only fails outright if every DataNode assigned to a block becomes unreachable simultaneously, which is rare enough that HDFS writes are considered reliable even on a cluster experiencing routine hardware churn.
Cricket analogy: Like a bowling attack continuing the innings with a replacement bowler when one pulls a hamstring mid-over, HDFS keeps a write going by swapping in a new DataNode rather than abandoning the whole innings.
Because HDFS pipelines require acknowledgment from every DataNode before a write is confirmed, write latency is bounded by the slowest node in the pipeline. A single consistently slow or overloaded DataNode can noticeably degrade write throughput cluster-wide, which is why monitoring per-DataNode latency, not just aggregate cluster throughput, matters for diagnosing performance issues.
- The NameNode only ever hands out metadata; actual file bytes flow directly between clients and DataNodes.
- Writes stream through a pipeline of DataNodes, with acknowledgments flowing back before data is considered durable.
- Reads prefer the closest replica (same node, then same rack, then off-rack) and verify checksums per chunk.
- The DFSClient library transparently handles pipeline construction, failover, and checksum verification.
- A failed DataNode mid-write is replaced in the pipeline rather than aborting the write outright.
- Blocks under-replicated after a mid-write failure are re-replicated automatically afterward.
- Write latency is bounded by the slowest DataNode in the pipeline, making per-node monitoring important.
Practice what you learned
1. Does file data ever pass through the NameNode during a read or write?
2. In the HDFS write path, when is a packet considered durably written?
3. How does the NameNode order the DataNode list it returns for a read request?
4. What happens if a DataNode in an active write pipeline fails?
5. What determines the write throughput of an HDFS pipeline?
Was this page helpful?
You May Also Like
HDFS Architecture
How HDFS organizes a cluster around a master/worker design so it can store and stream massive files reliably across commodity hardware.
NameNode and DataNodes
A closer look at HDFS's two core daemons: the NameNode that governs metadata and the DataNodes that physically store block data.
Blocks and Replication
How HDFS splits files into large blocks and replicates them across the cluster to guarantee durability and availability.
HDFS Commands
The essential hdfs dfs and hdfs dfsadmin commands for navigating, managing, and administering data in HDFS from the command line.
Related Reading
Related Study Notes in Programming
Browse all study notesApache Spark Study Notes
Programming · 30 topics
ProgrammingApache Flink Study Notes
Programming · 30 topics
ProgrammingSnowflake Study Notes
Programming · 30 topics
ProgrammingApache Airflow Study Notes
Programming · 30 topics
Programmingdbt (Data Build Tool) Study Notes
Programming · 30 topics
ProgrammingPowerShell Study Notes
Programming · 30 topics