Buffering
Buffering is the temporary storage of data or work between production and consumption to accommodate differences in processing rates, absorb interruptions, or improve transfer efficiency.
Logs and Telemetry
Buffering is the technique of temporarily storing data between processing stages to absorb differences in their processing rates.
- How it is used
- Absorbs input spikes and output slowdowns while moving observability events through Fluent Bit pipelines.
- How deep it goes
- Explains chunks, capacity, flush timing, memory versus filesystem storage, and window-retention trade-offs without exhaustive later detail.
- Examples
- Uses chunk handoff and hopping-window calculations over dummy events to illustrate retained and flushed data.
- What you can do after
- Operate/configure it by choosing storage, chunk capacity, flush behavior, and retention limits for expected destination and event-load conditions.
Go in Practice, Second Edition
Buffered I/O accumulates input or output in memory buffers so data can be processed or transferred in larger units.
- How it is used
- Uses buffering to stream file records, protect HTTP responses from partial template output, and improve network-log reliability.
- How deep it goes
- Works through line scanning and buffered template execution, including error handling and response-delay trade-offs; only sketches buffering for network logs.
- Examples
- Shows a scanner processing structured-log lines and a handler rendering a template into bytes.Buffer before responding.
- What you can do after
- Apply it via Go's bufio.NewScanner and bytes.Buffer to process line-delimited records and withhold template output until execution succeeds.
Read how Go in Practice, Second Edition explains buffering →
Kafka for Architects
Buffering temporarily stores data between producers and downstream consumers to absorb differences in processing rate or load.
- How it is used
- Deploys buffering as an intermediary in centralized log collection and delivery.
- How deep it goes
- Explains the log-transfer path and contrasts alternatives, but does not detail buffering internals or configuration.
- Examples
- Uses application log files, Fluentd agents, Kafka, Kafka Connect, Elasticsearch, and Kibana as the concrete pipeline.
- What you can do after
- Apply it via a Kafka-based log pipeline or compare it with direct, syslog, and alternative ingestion architectures.
Go in Action, Second Edition
Buffering is the technique of temporarily accumulating input or output data in memory to reduce the cost of individual transfers.
- How it is used
- Reduces memory pressure and system-call overhead when reading files, processing streams, or writing output.
- How deep it goes
- Explains buffering mechanics, bufio wrappers, line reads, peeking, writes, flushing, and practical conversion-tool integration.
- Examples
- Demonstrates line-by-line reading, non-consuming format detection, buffered writing, and JSON/XML conversion.
- What you can do after
- Apply buffering via Go's bufio package to process input incrementally and write output efficiently.
Build AI-Enhanced Web Apps
Data buffering temporarily stores incoming data chunks so they can be processed and delivered smoothly.
- How it is used
- Smooths incremental AI responses in web interfaces by managing arriving data before UI updates.
- How deep it goes
- Explains buffering within streaming architecture and chunked transfer encoding, but omits implementation code and low-level algorithms.
- Examples
- Uses a chunked text response and diagrams contrasting complete-response delivery with incremental streaming.
- What you can do after
- Apply it via Vercel AI SDK hooks and utilities to process streamed data and update a web UI smoothly.
Related concepts
- lockingconfigured by · Locking is a synchronization technique that restricts concurrent access to shared resources or protected operations.
- non-blocking I/Oused by · Non-blocking I/O is an input/output technique in which an operation returns control without requiring the initiating activity to wait for its completion.
- constraintaddresses · A constraint is a condition or rule that restricts the permissible states, values, actions, or solutions in a system or problem.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.