Streams, Piping, and Backpressure

Reading a 2 GB video with fs.readFile loads the whole file into memory before you can use a single byte. Streams let you process data piece by piece, as it arrives, using only a small amount of memory. Piping connects streams together, and backpressure stops a fast stream from flooding a slow one.

Definition

• Stream: an object that moves data in small pieces (chunks) over time, instead of all at once.
• Piping: connecting the output of one stream to the input of another, so data flows through automatically.
• Backpressure: a signal from a slow destination telling a fast source to pause until it catches up.

Analogy: think of moving water from a tank to a garden.

• Without streams: you fill one giant bucket with all the water, carry it over, and then pour. You need a bucket as big as the tank.
• With streams: you connect a pipe. Water flows a little at a time, and the garden gets it right away.
• Backpressure: if the garden can't soak it up fast enough, you close the tap a bit so it doesn't flood.

Why Streams?

const fs = require("fs");

// Without streams: the whole 2 GB file is loaded into memory
fs.readFile("video.mp4", (err, data) => {
  // data is a 2 GB Buffer
});

// With streams: read in 64 KB chunks
fs.createReadStream("video.mp4").on("data", (chunk) => {
  // chunk is about 64 KB
});

The Four Types of Streams

All streams are event emitters, so they report what happens through events.

Readable Streams

const fs = require("fs");

const readable = fs.createReadStream("data.txt", { encoding: "utf8" });

readable.on("data", (chunk) => console.log("Got", chunk.length, "characters"));
readable.on("end", () => console.log("No more data"));
readable.on("error", (err) => console.error("Failed:", err.message));

The Modern Way: for await...of

const fs = require("fs");

async function countBytes(path) {
  let total = 0;
  for await (const chunk of fs.createReadStream(path)) {
    total += chunk.length;
  }
  return total;
}

Readable streams are async iterable. The loop waits for each chunk, handles backpressure for you, and errors can be caught with try / catch.

Writable Streams

const fs = require("fs");

const writable = fs.createWriteStream("output.txt");

writable.write("Hello ");
writable.write("World\n");
writable.end("Done\n"); // write one last chunk and close

writable.on("finish", () => console.log("All data written"));
writable.on("error", (err) => console.error("Failed:", err.message));

Transform Streams

A transform stream receives chunks, changes them, and passes them on.

const { Transform } = require("stream");

const toUpperCase = new Transform({
  transform(chunk, encoding, callback) {
    callback(null, chunk.toString().toUpperCase()); // (error, output)
  },
});

process.stdin.pipe(toUpperCase).pipe(process.stdout);
// type "hello" -> prints "HELLO"

Piping

.pipe()

const fs = require("fs");
const zlib = require("zlib");

fs.createReadStream("logs.txt")
  .pipe(zlib.createGzip())
  .pipe(fs.createWriteStream("logs.txt.gz"));

Data flows from the file, through gzip compression, into the new file, one chunk at a time. .pipe() returns the destination, so you can chain it.

The problem with .pipe():

• Errors are not forwarded. You need an "error" listener on every stream.
• If one stream fails, the others are not closed, which can leak file handles and memory.

pipeline(): The Recommended Way

const fs = require("fs");
const zlib = require("zlib");
const { pipeline } = require("stream/promises");

async function compress(input, output) {
  try {
    await pipeline(
      fs.createReadStream(input),
      zlib.createGzip(),
      fs.createWriteStream(output),
    );
    console.log("Compressed");
  } catch (err) {
    console.error("Failed:", err.message);
  }
}

compress("logs.txt", "logs.txt.gz");

Backpressure

The Problem

Reading from a disk is usually much faster than writing to a slow network or a slow disk. If the source keeps pushing data, the extra chunks pile up in memory.

// Wrong: ignores backpressure
const readable = fs.createReadStream("huge.log");
const writable = fs.createWriteStream("copy.log");

readable.on("data", (chunk) => {
  writable.write(chunk); // return value ignored
});

// If writing is slower than reading,
// chunks pile up in memory until the process runs out of it

How Backpressure Works

Every stream has an internal buffer with a limit called highWaterMark (64 KB for file streams by default).

1. writable.write(chunk) returns true while the buffer is below the limit.
2. When the buffer goes over the limit, write() returns false. This means: "stop sending for now".
3. The source should pause.
4. When the buffer empties, the writable emits drain. The source can resume.

Handling Backpressure Manually

// With events
readable.on("data", (chunk) => {
  const ok = writable.write(chunk);
  if (!ok) {
    readable.pause(); // stop reading
    writable.once("drain", () => readable.resume()); // continue when ready
  }
});

readable.on("end", () => writable.end());

The Easy Way

You almost never need to do this by hand. pipe(), pipeline(), and for await...of all handle backpressure automatically.

// Backpressure handled for you
await pipeline(
  fs.createReadStream("huge.log"),
  fs.createWriteStream("copy.log"),
);

Note: highWaterMark is a threshold, not a hard limit. write() still accepts data after returning false, which is exactly why ignoring it can fill up memory.

Streams in an HTTP Server

On a Node.js server, req is a Readable stream and res is a Writable stream.

const http = require("http");
const fs = require("fs");
const { pipeline } = require("stream/promises");

http
  .createServer(async (req, res) => {
    if (req.url === "/video") {
      res.setHeader("Content-Type", "video/mp4");
      try {
        await pipeline(fs.createReadStream("movie.mp4"), res);
      } catch (err) {
        console.error("Stream failed:", err.message); // e.g. the client disconnected
      }
      return;
    }

    if (req.url === "/upload" && req.method === "POST") {
      try {
        await pipeline(req, fs.createWriteStream("upload.bin"));
        res.end("Uploaded");
      } catch (err) {
        res.statusCode = 500;
        res.end("Upload failed");
      }
      return;
    }

    res.statusCode = 404;
    res.end();
  })
  .listen(3000);

Each user uses only a few chunks of memory, so the server can stream a 2 GB video to many users at once. If a user's connection is slow, backpressure slows down reading the file for that user only.

Object Mode

By default, streams carry Buffers or strings. With objectMode: true, each chunk can be any JavaScript value, such as a database row.

const { Readable } = require("stream");

const users = Readable.from([
  { id: 1, name: "Asha" },
  { id: 2, name: "Ravi" },
]); // Readable.from() uses object mode by default

for await (const user of users) {
  console.log(user.name);
}

In object mode, highWaterMark counts objects (default 16), not bytes.

Summary

• Streams process data in chunks, so memory stays small and data is available immediately.
• Four types: Readable, Writable, Duplex, and Transform.
• Readable events: data, end, error. Readables are also async iterable with for await...of.
• Writable: write(), end(), and the drain and finish events.
• .pipe() connects streams, but doesn't forward errors or clean up.
• pipeline() from stream/promises handles errors, cleanup, and backpressure. Prefer it.
• Backpressure: when write() returns false, pause the source and resume on drain.
• pipe(), pipeline(), and for await...of handle backpressure automatically.
• On a server, req is Readable and res is Writable.

Interview Answer: What Are Streams and Backpressure in Node.js?

A stream moves data in small chunks instead of loading everything into memory. This lets Node.js handle huge files with a small, constant amount of memory, and start working on the data immediately. There are four types: Readable (like fs.createReadStream or an HTTP request), Writable (like fs.createWriteStream or an HTTP response), Duplex (like a TCP socket), and Transform, which changes data as it passes, like zlib.createGzip().

Piping connects streams so data flows from one to the next. I prefer pipeline() from stream/promises over .pipe(), because it forwards errors to one try / catch and destroys all streams if one fails, so nothing leaks.

Backpressure happens when the source is faster than the destination. Each writable has a buffer limit called highWaterMark. When the buffer is full, write() returns false, and the source should pause until the writable emits drain. If I ignore that, chunks pile up in memory and can crash the process. pipe(), pipeline(), and for await...of all handle backpressure automatically, so I only manage it by hand when calling write() myself.