liveBook · Manning

Backpressure

As explained in Secrets of the JavaScript Ninja, Third Edition

Backpressure is feedback from a consumer that regulates how quickly a producer supplies data based on the consumer's capacity or demand.

Backpressure lets a stream consumer influence how quickly its producer supplies data. In a pull stream, the consumer’s high watermark controls when the source is asked for more chunks. section 11.3.1 section 11.3.2

The practical tradeoff depends on the stream type:

  • A higher high watermark can buffer more while a sink is delayed, reducing the time it waits for data and potentially increasing throughput. section 11.3.2
  • For pull streams, the watermark controls when pull is called. Push streams may treat it only as a hint, so check how each stream implements it. section 11.3.2
Listing 11.8 Pulling calculations from a stream
import { setTimeout } from "node:timers/promises";
 
function createFibonacciStream(count: number) {
  let a = 0;
  let b = 1;
  let remaining = count;
  return new ReadableStream<number>({ #A
    pull(controller) { #A
      controller.enqueue(a); #A
      [a, b] = [b, a + b]; #A
      if (--remaining === 0) controller.close(); #A
    }, #A
  }); #A
}

const first10: number[] = [];
for await (const n of createFibonacciStream(10)) { #B
  first10.push(n); #B
  await setTimeout(0); #B
} #B

console.assert(
  first10.join(", ") ===
    "0, 1, 1, 2, 3, 5, 8, 13, 21, 34",
  "First 10 Fibonacci numbers",
);

A Fibonacci ReadableStream enqueues values from pull as an asynchronous consumer reads them, illustrating pull-driven production. section 11.3.1

Listing 11.9 Speeding up throughout by raising the high watermark
import { setTimeout } from "node:timers/promises";
 
const totalChunks = 100;
const flushInterval = 10;
 
function createSource() { #A
  let remaining = totalChunks; #A
  return new ReadableStream<number>({ #A
    async pull(controller) { #A
      await setTimeout(0); #A
      controller.enqueue(remaining); #A
      if (--remaining === 0) controller.close(); #A
    }, #A
  }); #A
} #A
 
async function timeWithHighWaterMark(
  highWaterMark: number,
) {
  let count = 0;
  const sink = new WritableStream<number>( #B
    { #B
      async write() { #B
        if (++count % flushInterval === 0) { #B
          await setTimeout(10); #B
        } #B
      }, #B
    }, #B
    { highWaterMark }, #B
  ); #B

  const start = performance.now(); #C
  await createSource().pipeTo(sink); #C
  return Math.round(performance.now() - start); #C
}

const time1 = await timeWithHighWaterMark(1);
const time10 = await timeWithHighWaterMark(10);

console.log(`highWaterMark 1: ${time1}ms`);
console.log(`highWaterMark 10: ${time10}ms`);
console.assert(
  time10 < time1,
  "Higher highWaterMark, higher throughput",
);

A delayed sink and configurable high watermark form a timing comparison of stream throughput under different buffering settings. section 11.3.2

How Secrets of the JavaScript Ninja, Third Edition covers it

How deep it goes
Explains proactive buffering, high-watermark effects on pull calls, and why push-stream behavior may be only a hint.
Examples
Demonstrates a delayed Fibonacci-stream consumer and a throughput timing comparison using a delayed sink.
What you can do after
Operate/configure Web Streams by adjusting a consumer’s high watermark to trade buffering against throughput, while checking push-stream behavior.
How it connects
Links streams with buffering, pull streams, push streams, producers, consumers, and high watermarks.

Also explained in

Compare how 10 books explain backpressure →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Secrets of the JavaScript Ninja, Third Edition. The section links point to the parts of the book it draws on.