liveBook · Manning

Data stream

As explained in Secrets of the JavaScript Ninja, Third Edition

A stream is an ordered queue-like data structure that represents values as they become available, including values that do not yet exist.

A video can start playing before it finishes downloading. The same idea helps with large files, generated text, and HTTP responses: process each piece as it arrives instead of waiting for everything. A stream is an ordered queue of values, including values that have not been produced yet. Those pieces are called chunks. section 11.2

A stream can be read with `for await…of`: each iteration waits for a chunk, handles it, and continues until the stream ends. That makes streams a kind of async iterator, but streams also support transformation, piping, and control over data flow. Merely collecting every chunk and processing it afterward gives up much of the memory and responsiveness benefit. section 11.2 section 11.2.1

Decode while an HTTP response arrives

A server writes one word every 100 ms; the response body can be consumed chunk by chunk rather than after the final word. Chunk boundaries are not character boundaries: the network may split a multibyte UTF-8 character. Use a `TextDecoder` with `{ stream: true }` when decoding successive byte chunks. section 11.2.1

Piping connects a readable source to a writable sink, optionally through a transform. `pipeThrough(new TextDecoderStream())` turns byte chunks into strings; `pipeTo(sink)` delivers the results. Do not assume one input chunk produces one output chunk. A transform may combine or split chunks. section 11.2.2

Listing 11.6 Piping a stream
const ninjaKanji = "忍者"; #A
 
const bytes = new TextEncoder().encode(ninjaKanji); #B
const splitAt = Math.floor( #C
  Math.random() * bytes.length, #C
); #C
const chunk1 = bytes.subarray(0, splitAt); #C
const chunk2 = bytes.subarray(splitAt); #C
 
const source = new ReadableStream({ #D
  start(controller) { #D
    controller.enqueue(chunk1); #D
    controller.enqueue(chunk2); #D
    controller.close(); #D
  }, #D
}); #D
 
const received: string[] = []; #E
const sink = new WritableStream<string>({ #E
  write(text) { #E
    received.push(text); #E
  }, #E
}); #E

await source #F
  .pipeThrough(new TextDecoderStream()) #F
  .pipeTo(sink); #F

console.assert(
  received.join("") === ninjaKanji,
  "TextDecoderStream correctly decodes across chunk boundaries",
);

The byte chunks may split a character, yet the decoder stream produces the original text for the sink. section 11.2.2

A push stream enqueues values without waiting for a consumer; a pull stream produces them in response to reads. Both look alike to the reader, but the choice can affect speed and memory use. Streams can also buffer queued chunks, so streaming alone does not guarantee a fixed memory footprint. section 11.3.1 section 11.4

Listing 11.8 Pulling calculations from a stream
import { setTimeout } from "node:timers/promises";
 
function createFibonacciStream(count: number) {
  let a = 0;
  let b = 1;
  let remaining = count;
  return new ReadableStream<number>({ #A
    pull(controller) { #A
      controller.enqueue(a); #A
      [a, b] = [b, a + b]; #A
      if (--remaining === 0) controller.close(); #A
    }, #A
  }); #A
}

const first10: number[] = [];
for await (const n of createFibonacciStream(10)) { #B
  first10.push(n); #B
  await setTimeout(0); #B
} #B

console.assert(
  first10.join(", ") ===
    "0, 1, 1, 2, 3, 5, 8, 13, 21, 34",
  "First 10 Fibonacci numbers",
);

A `ReadableStream` generates Fibonacci values through `pull(controller)` as an async loop consumes them. section 11.3.1

How Secrets of the JavaScript Ninja, Third Edition covers it

How deep it goes
Works through reading, incremental decoding, piping, and push-versus-pull behavior, while qualifying memory savings with buffering.
Examples
Uses a delayed HTTP text response, byte chunks split across characters, and a pull-driven Fibonacci sequence.
What it assumes
Assumes familiarity with async iterators and prior UTF-8 discussion; defines streams as queues before developing their mechanics.
What you can do after
Build it from scratch with ReadableStream, or apply Web stream APIs to decode, transform, and pipe arriving chunks.
How it connects
Connects async iterators, buffers, UTF-8 decoding, readable and writable streams, transforms, piping, and high watermarks.

Also explained in

Compare how 6 books explain data stream →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Secrets of the JavaScript Ninja, Third Edition. The section links point to the parts of the book it draws on.