liveBook · Manning

Pull stream

As explained in Secrets of the JavaScript Ninja, Third Edition

A pull stream is a stream that enqueues data in response to an indication that its consumer is ready to receive it.

A pull stream enqueues data when its consumer requests it, rather than producing data regardless of whether the consumer is ready. A stream can combine both behaviors, starting with push and later responding to pulls. section 11.2.1 section 11.3.1

In a ReadableStream, the pull method connects consumer demand to production:

  • The stream calls pull when a consumer tries to read a result. section 11.3.1
  • A high watermark guides how much the stream buffers; setting it to zero gives more direct flow, while a larger value can improve throughput at the cost of memory. section 11.3.2
  • Backpressure lets the consumer's buffering target affect the producer's behavior. section 11.3.2
Listing 11.8 Pulling calculations from a stream
import { setTimeout } from "node:timers/promises";
 
function createFibonacciStream(count: number) {
  let a = 0;
  let b = 1;
  let remaining = count;
  return new ReadableStream<number>({ #A
    pull(controller) { #A
      controller.enqueue(a); #A
      [a, b] = [b, a + b]; #A
      if (--remaining === 0) controller.close(); #A
    }, #A
  }); #A
}

const first10: number[] = [];
for await (const n of createFibonacciStream(10)) { #B
  first10.push(n); #B
  await setTimeout(0); #B
} #B

console.assert(
  first10.join(", ") ===
    "0, 1, 1, 2, 3, 5, 8, 13, 21, 34",
  "First 10 Fibonacci numbers",
);

A Fibonacci stream enqueues one value per pull and closes after producing the requested count; the consumer pauses between reads. section 11.3.1

Push and pull streams can look identical to consumers, but their implementation affects performance. Pull-based calculation can help batch work to avoid blocking the main thread. For XML parsing, push may suit processing whole files quickly, while pull may suit extracting information from large files with less memory. section 11.3.1

How Secrets of the JavaScript Ninja, Third Edition covers it

How deep it goes
Explains the pull method, buffering, and backpressure, including throughput and memory tradeoffs.
Examples
Demonstrates a Fibonacci stream and compares XML parsing libraries using push and pull approaches.
What it assumes
Assumes a consumer that reads stream results and contrasts pull behavior with push streams.
What you can do after
Build a ReadableStream that produces values on demand and closes after a set number of pulls.
How it connects
Connects pull streams with push streams, ReadableStream, backpressure, and Web Workers.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Secrets of the JavaScript Ninja, Third Edition. The section links point to the parts of the book it draws on.