liveBook · Manning

Stream processing

As explained in Secrets of the JavaScript Ninja, Third Edition

Streaming is the incremental processing or transfer of data as it becomes available rather than waiting for the complete data set.

A video can start playing before the whole file downloads. Streaming applies the same idea to other data: read or write small chunks as they become available instead of waiting for the complete result. This can reduce wait times and avoid holding a large file in memory at once. section 11.2 section 11.0

In JavaScript, a stream represents an ordered queue of values, including values that have not arrived yet. You can consume an HTTP request with `for await...of`: each iteration waits for a chunk, handles it, then waits for the next. Streams also support operations beyond iteration, including transformation and control over data flow. section 11.2 section 11.2.1

A response that arrives word by word

An HTTP server can write a word, pause, and write the next one while a client reads the response body as a stream. The client can display text as it arrives, but it must not assume that each received chunk matches a word the server wrote: the network may split chunks differently. section 11.2.1

Two traps can erase the benefit or corrupt the result:

  • Collecting every chunk and decoding only after the stream ends delays the work and requires memory for both the chunks and their combined buffer. section 11.2.1
  • Decoding each buffer independently can break a UTF-8 character split across chunks. Keep decoder state across chunks with `TextDecoder.decode(chunk, { stream: true })`. section 11.1.1 section 11.2.1

To produce a stream yourself, create a `ReadableStream`, enqueue values as they become available, and close it when production ends. The consumer can begin reading before the producer has enqueued every value. section 11.2.1

Listing 11.5 Creating a ReadableStream
import { setTimeout } from "node:timers/promises";
 
const ninjas = [
  "Leonardo",
  "Donatello",
  "Raphael",
  "Michelangelo",
];
 
const source = new ReadableStream<string>({
  async start(controller) { #A
    for (const ninja of ninjas) { #A
      controller.enqueue(ninja); #A
      await setTimeout(100); #A
    } #A
    controller.close(); #A
  },
});

const received: string[] = []; #B
for await (const ninja of source) { #B
  received.push(ninja); #B
} #B

console.assert(
  received.length === 4,
  "Four ninjas were received",
);

A `ReadableStream` producer enqueues delayed values while an async loop consumes them. section 11.2.1

How Secrets of the JavaScript Ninja, Third Edition covers it

How deep it goes
Explains chunk-by-chunk consumption, production, aggregation costs, and incremental text decoding; only previews transformation and flow control.
Examples
Uses a chunked HTTP request, a word-by-word HTTP response, and a ReadableStream that enqueues delayed values.
What it assumes
Assumes familiarity with async iteration for reading streams and introduces chunked data handling through buffers and HTTP examples.
What you can do after
Build it from scratch with ReadableStream and consume incoming chunks while preserving decoder state for UTF-8 text.
How it connects
Connects streams to buffers, async iterators, HTTP requests and responses, UTF-8 decoding, and ReadableStream.

Also explained in

Compare how 25 books explain stream processing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Secrets of the JavaScript Ninja, Third Edition. The section links point to the parts of the book it draws on.