liveBook · Manning

Pipeline

As explained in JavaScript in Depth

A sequence of connected stream operations through which data flows from a source toward a destination.

A pipeline connects a producer to a consumer through zero or more transforms. Data flows through the chain as chunks, while either the producer pushes data eagerly or the consumer pulls it lazily. The data type is unrestricted: bytes, objects, strings, and numbers can all pass through. section 10.1

The main correctness rule is to preserve backpressure rather than pushing by hand. Every buffer has its own capacity and signal; built-in piping coordinates those signals across the chain. Ignoring them lets data accumulate wherever downstream processing is slower. section 10.3.3 section 10.9

Pipeline behavior changes with the streaming model and its boundaries:

  • Node.js streams report failures through error events; pipeline() propagates the failure, destroys the chain, and delivers it to a callback or rejected promise. section 10.6.1
  • Web Streams use rejected promises; pipeThrough() inserts transforms, while pipeTo() terminates the chain and manages backpressure, errors, and cleanup. section 10.4.4 section 10.6.1
  • Mixing Node.js streams, Web Streams, and async iterables crosses different error and scheduling models, adding adapters, buffers, and per-chunk overhead. section 10.6.4 section 10.6

Chaining transforms

const transformed = readable .pipeThrough(transformA) .pipeThrough(transformB); await transformed.pipeTo(writable); This keeps the transforms in one Web Streams pipeline and lets the piping operations coordinate completion, errors, cleanup, and backpressure. section 10.4.4

Figure 10.1 The fundamental flow of data through a stream is the same regardless of the specific implementation. A producer provides chunks of data that flow through a series of transforms and on to a consumer.

Use the flow diagram to inspect the stable pipeline shape: producer, transforms, and consumer remain conceptually distinct across stream implementations. section 10.1

How JavaScript in Depth covers it

How deep it goes
Explains pipeline roles, pacing, buffers, errors, conversions, and per-chunk costs, but defers general runtime execution pipelines.
Examples
Demonstrates chained Web Stream transforms, buffered Node.js transforms, and conversions between Node.js and Web Streams.
What it assumes
Assumes familiarity with streams and asynchronous errors, while defining producers, consumers, transforms, push, and pull from first principles.
What you can do after
Build it from scratch by chaining producers, transforms, and consumers while preserving backpressure and handling model-specific errors.
How it connects
Connects producers, consumers, transforms, backpressure, error propagation, Web Streams, Node.js streams, and async iterables.

Also explained in

Compare how 63 books explain pipeline →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of JavaScript in Depth. The section links point to the parts of the book it draws on.