Buffering
Buffering is the temporary storage of data between processing or transfer steps, often to accommodate differences in rates or reduce operation costs.
Secrets of the JavaScript Ninja, Third Edition
The temporary storage of produced data before it is consumed.
- How it is used
- Holds raw bytes for later interpretation and queues stream chunks so a consumer need not wait for every production step.
- How deep it goes
- Works through byte-length checks and ReadableStream queue behavior, including defaults, high-water-mark limits, and the memory–throughput tradeoff.
- Examples
- Uses an HTTP request body, a PNG header, a pull stream producing numbers, and a sink with periodic delays.
- What you can do after
- Operate/configure it by checking buffered byte lengths and adjusting a stream’s high watermark to balance waiting against memory use.
Read how Secrets of the JavaScript Ninja, Third Edition explains buffering →
Logs and Telemetry
Buffering is the technique of temporarily storing data between processing stages to absorb differences in their processing rates.
- How it is used
- Absorbs input spikes and output slowdowns while moving observability events through Fluent Bit pipelines.
- How deep it goes
- Explains chunks, capacity, flush timing, memory versus filesystem storage, and window-retention trade-offs without exhaustive later detail.
- Examples
- Uses chunk handoff and hopping-window calculations over dummy events to illustrate retained and flushed data.
- What you can do after
- Operate/configure it by choosing storage, chunk capacity, flush behavior, and retention limits for expected destination and event-load conditions.
Go in Practice, Second Edition
Buffered I/O accumulates input or output in memory buffers so data can be processed or transferred in larger units.
- How it is used
- Uses buffering to stream file records, protect HTTP responses from partial template output, and improve network-log reliability.
- How deep it goes
- Works through line scanning and buffered template execution, including error handling and response-delay trade-offs; only sketches buffering for network logs.
- Examples
- Shows a scanner processing structured-log lines and a handler rendering a template into bytes.Buffer before responding.
- What you can do after
- Apply it via Go's bufio.NewScanner and bytes.Buffer to process line-delimited records and withhold template output until execution succeeds.
Read how Go in Practice, Second Edition explains buffering →
Kafka for Architects
Buffering temporarily stores data between producers and downstream consumers to absorb differences in processing rate or load.
- How it is used
- Deploys buffering as an intermediary in centralized log collection and delivery.
- How deep it goes
- Explains the log-transfer path and contrasts alternatives, but does not detail buffering internals or configuration.
- Examples
- Uses application log files, Fluentd agents, Kafka, Kafka Connect, Elasticsearch, and Kibana as the concrete pipeline.
- What you can do after
- Apply it via a Kafka-based log pipeline or compare it with direct, syslog, and alternative ingestion architectures.
JavaScript in Depth
An internal data structure that temporarily stores stream data between production and consumption.
- How it is used
- Paces stream production against consumption by temporarily retaining chunks and participating in backpressure.
- How deep it goes
- Explains paused-stream accumulation, per-stage highWaterMark thresholds, and how ignored signals let buffers grow.
- Examples
- Uses paused Readables, Writable write() and drain signals, multi-buffer pipelines, and Web Streams pull cycles.
- What you can do after
- Diagnose chunk accumulation when a Readable is paused or a producer ignores backpressure signals.
Go in Action, Second Edition
Buffering is the technique of temporarily accumulating input or output data in memory to reduce the cost of individual transfers.
- How it is used
- Reduces memory pressure and system-call overhead when reading files, processing streams, or writing output.
- How deep it goes
- Explains buffering mechanics, bufio wrappers, line reads, peeking, writes, flushing, and practical conversion-tool integration.
- Examples
- Demonstrates line-by-line reading, non-consuming format detection, buffered writing, and JSON/XML conversion.
- What you can do after
- Apply buffering via Go's bufio package to process input incrementally and write output efficiently.
Systems Programming with Zig
Buffering is the temporary storage of input or output data before it is processed or transferred.
- How it is used
- Buffers output to avoid sending each write to a file immediately, where writing data causes latency.
- How deep it goes
- Defines buffering and flushing, explains newline or byte-count triggers, and contrasts implicit runtime behavior with explicit Zig control.
- Examples
- Uses a Zig stdout writer with a 1,024-byte buffer, a newline-terminated message, and an explicit flush.
- What you can do after
- Operate/configure buffering by supplying a stdout buffer and flushing the writer explicitly.
Build AI-Enhanced Web Apps
Data buffering temporarily stores incoming data chunks so they can be processed and delivered smoothly.
- How it is used
- Smooths incremental AI responses in web interfaces by managing arriving data before UI updates.
- How deep it goes
- Explains buffering within streaming architecture and chunked transfer encoding, but omits implementation code and low-level algorithms.
- Examples
- Uses a chunked text response and diagrams contrasting complete-response delivery with incremental streaming.
- What you can do after
- Apply it via Vercel AI SDK hooks and utilities to process streamed data and update a web UI smoothly.
Also covered in
- 100 SQL Server Mistakes and How to Avoid Them
- Python Concurrency with asyncio
- Learn Concurrent Programming with Go
- Fundamentals of CybersecurityIn early access (MEAP)
- 100 C++ Mistakes and How to Avoid Them
- Akka in Action, Second Edition
- Learn Go with Pocket-Sized Projects
- ASP.NET Core in Action, 3e
- Grokking Concurrency
- Designing AI SystemsIn early access (MEAP)
- Kotlin in Action, 2E
- 100 Go Mistakes and How to Avoid Them
What to know first
- throughputThroughput is the rate at which a system processes or completes work, data, or requests over a specified period.
- concurrencyConcurrency is the structuring or execution of multiple computational activities so their progress overlaps in time, whether or not they run simultaneously.
- standard outputStandard output is a process's conventional stream for writing ordinary program output, which may be directed to a terminal, file, pipe, or other destination.
- JSONJSON is a language-independent text-based data-interchange format that represents structured values using objects, arrays, strings, numbers, booleans, and null.
Related concepts
- bufioimplemented by · bufio is Go’s standard-library package providing buffered input and output wrappers and utilities for incrementally processing streams.
- chunk sizeconfigured by · Chunk size is the amount of data or content assigned to each unit when a larger source is divided into chunks for processing, storage, transfer, or retrieval.
Questions about buffering
What is buffering?
Buffering is the temporary storage of data between processing or transfer steps, often to accommodate differences in rates or reduce operation costs.
Which Manning books explain buffering?
Secrets of the JavaScript Ninja, Third Edition, Logs and Telemetry, Go in Practice, Second Edition, Kafka for Architects, JavaScript in Depth, Go in Action, Second Edition, Systems Programming with Zig and Build AI-Enhanced Web Apps.
What should I know before learning buffering?
Manning books that teach buffering usually introduce throughput, concurrency, standard output and JSON first.
What is buffering a kind of?
Buffering is a kind of Uint8Array. Uint8Array is a JavaScript typed-array constructor whose instances store elements as unsigned 8-bit integers.
What is buffering also called?
Buffering is also called buffer.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.







