10 Streams
This chapter covers:
- Comparing the three common streaming architectures in JavaScript
- Distinguishing push streams from pull streams
- Defining and tracing how backpressure works in each model
- Measuring the per-chunk-of-data cost of each model
- Identifying the patterns that quietly undermine stream performance
The previous chapters have each focused on a single aspect of JavaScript's hidden machinery: how data types work, how functions execute, how errors propagate, how the scheduling queues interact, how the garbage collector competes for the main thread, and so on. Streams are one example of a higher-order mechanism where all of these converge. A stream pipeline creates closures that capture scope, schedules callbacks through event emitters or promises, allocates objects for every chunk of data, and propagates errors across asynchronous boundaries. Every chapter in this book so far describes something that a stream pipeline does continuously, under load, for the lifetime of the data flow.
A stream is a way of moving data through an application one chunk at a time rather than all at once. Instead of loading an entire file, response, or query result into memory and then act on it, a stream hands your code one piece, lets you process it, and moves on to the next. Work can begin before all the data has arrived, and only a small working set is held in memory at any moment no matter how large the full dataset is.