Streams & Buffers

Concept#

The beginner framing: reading an entire 2GB video file into memory before doing anything with it is wasteful and slow, streams let you process data in small chunks as it arrives, without ever holding the whole thing in memory at once.

The precise mental model: a stream is an EventEmitter-based (see EventEmitter & Async Patterns) abstraction for reading or writing data incrementally. A Buffer is the raw binary data structure streams pass around chunk by chunk, a fixed-size block of memory, allocated outside the normal V8 JS heap, for efficiently handling binary data (file contents, network packets) that doesn't map naturally onto JS strings.

const fs = require("fs");
 
const buf = Buffer.from("hello", "utf8");
console.log(buf); // <Buffer 68 65 6c 6c 6f>
console.log(buf.toString("utf8")); // "hello"

Four stream types#

Backpressure: the mechanism that makes streaming actually work#

const writable = fs.createWriteStream("output.txt");
 
const canContinue = writable.write(chunk); // returns a BOOLEAN
if (!canContinue) {
  // internal buffer has reached highWaterMark, PAUSE producing more
  producer.pause();
}
 
writable.on("drain", () => {
  // buffer has room again, SAFE to resume
  producer.resume();
});

write() returns false once the writable's internal buffer reaches its highWaterMark (a size threshold, default 16KB for byte streams). That's the writable stream telling the producer "slow down, I can't keep up." The 'drain' event fires once the buffer has drained back down and there's room again, the producer's signal to resume. Without honoring this signal, a fast producer writing to a slow destination just keeps buffering data in memory indefinitely, which is exactly the memory-blowup streams exist to prevent.

const ok = writable.write(chunk1);
// ok === true, buffer has room, keep writing

.pipe(): automating backpressure across an entire chain#

fs.createReadStream("huge-file.txt")
  .pipe(zlib.createGzip())            // Transform stream
  .pipe(fs.createWriteStream("huge-file.txt.gz")); // Writable

Manually managing write()'s return value and 'drain' events gets complex fast, especially across multiple chained stages. .pipe() automates the entire pause/resume cycle, and critically, propagates it backward through the whole chain: if the final writable's buffer fills up, .pipe() automatically pauses the stage feeding it, which (since that stage is itself piped from upstream) automatically pauses the stage feeding that, and so on, all the way back to the original readable source.

readable.pipe(transform).pipe(writable);

Object-mode streams: counting objects, not bytes#

const { Readable } = require("stream");
 
const objStream = new Readable({
  objectMode: true,
  highWaterMark: 16, // 16 OBJECTS, not 16KB of bytes
  read() {}
});

In object mode, highWaterMark counts individual objects pushed into the stream, regardless of each object's actual size, a stream holding sixteen 10-byte objects and a stream holding sixteen 10-megabyte objects both hit the same highWaterMark of 16, since the count is objects, not bytes.

const stream = new Readable({
  objectMode: true,
  highWaterMark: 16, // default for object mode, COUNTS OBJECTS
});

Try It#

Predict what happens before checking the solution.

const writable = fs.createWriteStream("out.txt", { highWaterMark: 16 });
 
for (let i = 0; i < 1000; i++) {
  const ok = writable.write(`line ${i}\n`);
  if (!ok) {
    console.log(`backpressure hit at iteration ${i}`);
    break;
  }
}

Given a tiny highWaterMark of 16 bytes, roughly how many iterations run before write() returns false?

Implement It Yourself#

Build a minimal backpressure-aware writer, without using real streams, to internalize the write/drain mechanism:

function createBackpressureAwareSink(highWaterMark = 5) {
  const buffer = [];
  let draining = false;
  const drainListeners = [];
 
  return {
    write(item) {
      buffer.push(item);
      if (buffer.length >= highWaterMark) {
        return false; // signal: PAUSE
      }
      return true; // signal: keep going
    },
    onDrain(fn) {
      drainListeners.push(fn);
    },
    // simulates the sink slowly consuming buffered items

This is the exact shape of real backpressure: a boolean return value signaling "pause," and a 'drain'-equivalent callback signaling "resume," decoupled from each other so the producer can react asynchronously.

Under the Hood#

Streams are built on EventEmitter (see EventEmitter & Async Patterns), 'data', 'end', 'drain', and 'error' are all just events dispatched via the same .on()/.emit() mechanism covered there, including the same special-cased crash-on-unhandled-'error' behavior. And every stream callback, whether it's a 'data' handler or a write() callback, ultimately runs as part of the event loop's phases (see Event Loop & libuv), typically surfacing through the poll phase once the underlying I/O completes.

Common Mistakes#

1. Ignoring write()'s return value#

for (const chunk of hugeArrayOfChunks) {
  writable.write(chunk); // ❌ never checks the return value or waits for 'drain'
}

This defeats backpressure entirely, a fast producer just keeps calling write() regardless of whether the destination can keep up, causing the internal buffer (and memory usage) to grow unboundedly.

2. Manually managing multi-stage pipelines instead of using .pipe()#

readable.on("data", (chunk) => {
  const ok = writable.write(chunk);
  if (!ok) readable.pause(); // ❌ error-prone to get right across MULTIPLE chained stages
});
writable.on("drain", () => readable.resume());

.pipe() (or the newer stream.pipeline() for chains with proper error propagation) already implements this correctly, including backward propagation across multiple stages, reimplementing it manually is easy to get subtly wrong, especially with more than two stages.

3. Confusing object-mode's highWaterMark with a byte count#

new Readable({ objectMode: true, highWaterMark: 16 })
// this is 16 OBJECTS, not 16 bytes or 16KB, a common misreading

In object mode, the number always refers to object count, not size, assuming otherwise leads to wildly incorrect memory expectations, especially with large objects.

Best Practices#

• Always check write()'s return value when writing manually (not via .pipe()), and pause the producer until 'drain' fires.
• Prefer .pipe() or stream.pipeline() over manual 'data'-event handling whenever chaining multiple stream stages, it's correct by construction and handles error propagation better (pipeline() in particular cleans up all streams properly on error, which raw .pipe() chains don't do automatically).
• Tune highWaterMark deliberately for your actual data shape, a very small value causes excessive backpressure churn, a very large value defeats the memory-efficiency purpose of streaming in the first place.

Performance Tips#

• Streaming a large file instead of reading it fully into memory (fs.readFileSync) keeps memory usage roughly constant regardless of file size, this is the entire performance case for streams over whole-file reads.
• Object-mode streams have real per-object overhead, for large volumes of small objects, consider whether batching multiple items per pushed object reduces overhead versus pushing each one individually.