Streams in Node.js

Analogy categoryAnalogies

Introduction

Streams are Node.js's abstraction for working with data incrementally, as a sequence of chunks, rather than loading an entire resource into memory at once. This is essential for efficiently handling large files, network sockets, and other I/O sources. Node.js provides four fundamental stream types: Readable (a source of data, e.g., fs.createReadStream), Writable (a destination for data, e.g., fs.createWriteStream), Duplex (both readable and writable, e.g., a TCP socket), and Transform (a duplex stream that modifies data as it passes through, e.g., zlib.createGzip).

🏏

Cricket analogy: Streams are like watching a match ball-by-ball on live commentary rather than waiting for the entire day's play to finish before hearing any of it; a Readable stream is the live feed itself, a Writable stream is the scoreboard updating in real time, a Duplex stream is a two-way radio between commentator and stats desk, and a Transform stream is the analyst converting raw ball data into instant graphics.

Syntax

javascript

const fs = require('fs');
const zlib = require('zlib');

const readStream = fs.createReadStream('input.txt');
const gzipTransform = zlib.createGzip();
const writeStream = fs.createWriteStream('input.txt.gz');

// pipe() connects streams and automatically manages backpressure
readStream
  .pipe(gzipTransform)
  .pipe(writeStream)
  .on('finish', () => console.log('Compression complete'));

readStream.on('error', (err) => console.error('Read error:', err));
writeStream.on('error', (err) => console.error('Write error:', err));

Explanation

The pipe() method reads chunks (Buffers, by default) from a Readable stream and writes them to a Writable stream, emitting 'data' events internally and handling flow control automatically. Backpressure occurs when a Writable destination can't accept data as fast as the Readable source produces it — write() returns false when its internal buffer exceeds the highWaterMark. pipe() respects this signal by pausing the Readable source until the Writable emits 'drain', preventing unbounded memory growth. Writing streams manually without pipe() requires handling this signal yourself.

🏏

Cricket analogy: pipe() is like a relay of fielders passing the ball from boundary to keeper automatically, matching pace so no one drops it; backpressure is when the keeper's hands are full and signals 'hold' -- pipe() makes the fielder wait until the keeper is ready ('drain') rather than piling balls up, whereas manual relaying without pipe() means you must watch for that signal yourself.

Example

javascript

const { Transform } = require('stream');

// A custom Transform stream that uppercases text chunks
class UppercaseTransform extends Transform {
  _transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback(); // signal this chunk is processed, ready for more
  }
}

const fs = require('fs');
const upper = new UppercaseTransform();

fs.createReadStream('input.txt', { highWaterMark: 16 * 1024 })
  .pipe(upper)
  .pipe(fs.createWriteStream('output.txt'))
  .on('finish', () => console.log('Done writing output.txt'));

// Manually handling backpressure without pipe()
const readable = fs.createReadStream('input.txt');
const writable = fs.createWriteStream('output-manual.txt');

readable.on('data', (chunk) => {
  const canContinue = writable.write(chunk);
  if (!canContinue) {
    readable.pause();
    writable.once('drain', () => readable.resume());
  }
});
readable.on('end', () => writable.end());

Output

The first pipeline reads input.txt in chunks (default highWaterMark of 64KB, overridden here to 16KB), transforms each chunk to uppercase via _transform(), and writes the result to output.txt, logging 'Done writing output.txt' once finished — all without loading the whole file into memory. The manual example demonstrates the same backpressure logic pipe() handles automatically: if writable.write() returns false, the code pauses the readable stream and resumes it only after 'drain' fires, keeping memory usage bounded even for very large files.

🏏

Cricket analogy: The first pipeline reads a day's raw ball-by-ball data in small overs-sized chunks, converts each entry to a standardized uppercase format via a transform step, and writes the polished result to the official scorecard file, logging completion -- all without loading the entire match archive into memory; the manual example mirrors this backpressure handling, pausing the feed when the scorecard writer signals full and resuming only after it drains.

Key Takeaways

The four stream types are Readable, Writable, Duplex, and Transform.
Streams process data in chunks, enabling memory-efficient handling of large files and network data.
pipe() connects streams and automatically manages backpressure between them.
Backpressure happens when write() returns false; you should pause the source and resume on 'drain'.
Custom Transform streams implement _transform(chunk, encoding, callback) to modify data in-flight.
Always attach 'error' listeners on each stream, since errors don't propagate automatically through pipe() chains.