Introduction
Streams are Node.js's abstraction for working with data incrementally, as a sequence of chunks, rather than loading an entire resource into memory at once. This is essential for efficiently handling large files, network sockets, and other I/O sources. Node.js provides four fundamental stream types: Readable (a source of data, e.g., fs.createReadStream), Writable (a destination for data, e.g., fs.createWriteStream), Duplex (both readable and writable, e.g., a TCP socket), and Transform (a duplex stream that modifies data as it passes through, e.g., zlib.createGzip).
Cricket analogy: Streams are like watching a match ball-by-ball on live commentary rather than waiting for the entire day's play to finish before hearing any of it; a Readable stream is the live feed itself, a Writable stream is the scoreboard updating in real time, a Duplex stream is a two-way radio between commentator and stats desk, and a Transform stream is the analyst converting raw ball data into instant graphics.
Syntax
const fs = require('fs');
const zlib = require('zlib');
const readStream = fs.createReadStream('input.txt');
const gzipTransform = zlib.createGzip();
const writeStream = fs.createWriteStream('input.txt.gz');
// pipe() connects streams and automatically manages backpressure
readStream
.pipe(gzipTransform)
.pipe(writeStream)
.on('finish', () => console.log('Compression complete'));
readStream.on('error', (err) => console.error('Read error:', err));
writeStream.on('error', (err) => console.error('Write error:', err));Explanation
The pipe() method reads chunks (Buffers, by default) from a Readable stream and writes them to a Writable stream, emitting 'data' events internally and handling flow control automatically. Backpressure occurs when a Writable destination can't accept data as fast as the Readable source produces it — write() returns false when its internal buffer exceeds the highWaterMark. pipe() respects this signal by pausing the Readable source until the Writable emits 'drain', preventing unbounded memory growth. Writing streams manually without pipe() requires handling this signal yourself.
Cricket analogy: pipe() is like a relay of fielders passing the ball from boundary to keeper automatically, matching pace so no one drops it; backpressure is when the keeper's hands are full and signals 'hold' -- pipe() makes the fielder wait until the keeper is ready ('drain') rather than piling balls up, whereas manual relaying without pipe() means you must watch for that signal yourself.
Example
const { Transform } = require('stream');
// A custom Transform stream that uppercases text chunks
class UppercaseTransform extends Transform {
_transform(chunk, encoding, callback) {
this.push(chunk.toString().toUpperCase());
callback(); // signal this chunk is processed, ready for more
}
}
const fs = require('fs');
const upper = new UppercaseTransform();
fs.createReadStream('input.txt', { highWaterMark: 16 * 1024 })
.pipe(upper)
.pipe(fs.createWriteStream('output.txt'))
.on('finish', () => console.log('Done writing output.txt'));
// Manually handling backpressure without pipe()
const readable = fs.createReadStream('input.txt');
const writable = fs.createWriteStream('output-manual.txt');
readable.on('data', (chunk) => {
const canContinue = writable.write(chunk);
if (!canContinue) {
readable.pause();
writable.once('drain', () => readable.resume());
}
});
readable.on('end', () => writable.end());Output
The first pipeline reads input.txt in chunks (default highWaterMark of 64KB, overridden here to 16KB), transforms each chunk to uppercase via _transform(), and writes the result to output.txt, logging 'Done writing output.txt' once finished — all without loading the whole file into memory. The manual example demonstrates the same backpressure logic pipe() handles automatically: if writable.write() returns false, the code pauses the readable stream and resumes it only after 'drain' fires, keeping memory usage bounded even for very large files.
Cricket analogy: The first pipeline reads a day's raw ball-by-ball data in small overs-sized chunks, converts each entry to a standardized uppercase format via a transform step, and writes the polished result to the official scorecard file, logging completion -- all without loading the entire match archive into memory; the manual example mirrors this backpressure handling, pausing the feed when the scorecard writer signals full and resuming only after it drains.
Key Takeaways
- The four stream types are Readable, Writable, Duplex, and Transform.
- Streams process data in chunks, enabling memory-efficient handling of large files and network data.
- pipe() connects streams and automatically manages backpressure between them.
- Backpressure happens when write() returns false; you should pause the source and resume on 'drain'.
- Custom Transform streams implement _transform(chunk, encoding, callback) to modify data in-flight.
- Always attach 'error' listeners on each stream, since errors don't propagate automatically through pipe() chains.
Practice what you learned
1. Which stream type both reads and modifies data as it passes through, such as zlib.createGzip()?
2. What does it mean when writable.write(chunk) returns false?
3. How does pipe() handle backpressure between a Readable and a Writable stream?
4. In a custom Transform stream, what must _transform(chunk, encoding, callback) do to emit output and signal readiness for the next chunk?
Was this page helpful?
You May Also Like
The fs Module in Node.js
Learn how to read, write, and manage files using Node's built-in fs module with sync, async, and promise-based APIs.
The events Module and EventEmitter
Learn how Node's EventEmitter class enables the observer pattern used throughout the Node.js core APIs.
Node.js Performance Optimization
Learn how to scale Node.js apps with the cluster module, avoid blocking the event loop, and apply caching strategies.
Node.js Architecture and the Event Loop
Understand Node.js's single-threaded, non-blocking architecture and how the event loop phases process callbacks.
Related Reading
Related Study Notes in Web Development
Browse all study notesWebSockets Study Notes
Web Development · 30 topics
Web DevelopmentWebAssembly Study Notes
WebAssembly · 30 topics
Web DevelopmentgRPC Study Notes
Protocol Buffers · 30 topics
Web DevelopmentSpring Boot Study Notes
Java · 30 topics
Web DevelopmentFlask Study Notes
Python · 30 topics
Web DevelopmentDjango Study Notes
Python · 30 topics