11 Buffers and streams
This chapter covers
- Using buffers for efficient memory usage
- How streaming reduces wait times
- Why JavaScript has two stream APIs
In the early days of web development, JavaScript’s job was simple: wait for a user to click a button, then clear a form input. Today, though, JavaScript is a true general-purpose language, used for processing multi-gigabyte video uploads and parsing million-line CSV files. Conventional JavaScript arrays are poorly suited to handling data with that kind of scale. We need to get a little closer to the metal.
The solution is two concepts: buffers and streams. Buffers give JavaScript the ability to read raw binary data rather than transforming those bytes into JavaScript’s relatively complex primitive types. Streams let JavaScript process data over time in small pieces called chunks. With the help of buffers and streams, JavaScript can process a 100GB file using only a few megabytes of RAM.
Buffers and streams were first introduced in Node.js, but today they’re standardized as part of the Web API built into browsers, servers, and all other modern JavaScript runtimes. In situations where performance is critical, whether rendering 3D games in the browser with WebGL or displaying output from an LLM as soon as it’s emitted, these APIs are essential.
We’ll start our journey with buffers, showing you how to work with raw binary data. Then we’ll dive into streaming, the art of reading and writing data as a series of chunks.