Stream processing
Stream processing is the incremental processing of data items as they become available, without requiring the complete dataset to be present first.
Functional Programming in Scala, Second Edition
Stream processing is the incremental computation of results from sequences of data as elements become available.
- How it is used
- Frames efficient external I/O as composable incremental transformations rather than monolithic loops.
- How deep it goes
- Develops Pull and Stream internals, pipes, effects, error handling, and scoped resource finalization.
- Examples
- Uses file temperature conversion, line counting, infinite integers, normalization pipes, and resource-managed file reading and writing.
- What you can do after
- Build it from scratch with Pull, Stream, Pipe, effects, error handlers, and resource finalizers.
Read how Functional Programming in Scala, Second Edition explains stream processing →
The Joy of JavaScript
A stream is a sequence of one or more values or events that can be consumed incrementally over time.
- How it is used
- Unifies incremental processing of collections, file chunks, and recurring asynchronous events under a sequence-of-values model.
- How deep it goes
- Works through async file iteration, a push-capable array, custom Observables, and composed operators; treats Observable as a proposal.
- Examples
- Uses chunked blockchain-file validation, an event-emitting array, periodic random numbers, and number-transforming Observable pipelines.
- What you can do after
- Build from scratch a small Observable operator pipeline, or apply async iteration to process and validate file blocks incrementally.
Apache Kafka in Action
Stream processing is the continuous computation of results over records as they arrive.
- How it is used
- Enables near-real-time analysis and responses to changes in data flowing through Kafka.
- How deep it goes
- Explains event-at-a-time processing and its contrast with batch results, while deferring implementation details.
- Examples
- Uses an online store’s product-price changes, customer segments, website updates, notifications, and sales alerts.
- What you can do after
- Evaluate/understand when near-real-time event processing suits a business decision better than scheduled batch processing.
Read how Apache Kafka in Action explains stream processing →
Build AI-Enhanced Web Apps
Streaming behavior is the incremental delivery or processing of output as portions become available.
- How it is used
- Uses streaming to make AI chat responses visible during generation and reduce perceived waiting time.
- How deep it goes
- Explains the client-server flow, asynchronous chunks, buffering, and error handling, then implements streaming through SDK utilities and hooks.
- Examples
- Demonstrates Gemini text streaming, a React chat API and client, and a weather-data chain updating a streamable UI.
- What you can do after
- Apply streaming via the Vercel AI SDK to deliver incremental model output in a React chat interface.
Read how Build AI-Enhanced Web Apps explains stream processing →
100 Java Mistakes and How to Avoid Them
The Java Stream API provides pipelines for processing sequences of elements through intermediate and terminal operations.
- How it is used
- Processes sequences through fluent transformation and result-producing pipelines while replacing some manual loops and index handling.
- How deep it goes
- Explains lazy execution, side-effect risks, single consumption, and reduce contracts, including sequential-versus-parallel failure modes.
- Examples
- Demonstrates predicate partitioning, conditional skip, repeated consumption, hash-code reduction, and inclusive numeric ranges.
- What you can do after
- Apply it via Java Stream API pipelines while avoiding side effects, reuse, incompatible reductions, and unsupported null handling.
Read how 100 Java Mistakes and How to Avoid Them explains stream processing →
Kafka for Architects
An event stream is a continuously available sequence of records representing events produced by a system.
- How it is used
- Positions continuous event transformation between Kafka producers and downstream consumers when low-latency results matter.
- How deep it goes
- Defines continuous processing and explains routing, enrichment, and state at an architectural level, without detailing streaming-programming implementations.
- Examples
- Uses address-based routing and order enrichment with customer data to illustrate event transformation.
- What you can do after
- Evaluate/understand it as a low-latency processing layer and choose where filtering, joins, and aggregation belong.
LLMs in Production
Streaming is the incremental retrieval and processing of a dataset without loading the entire dataset into memory.
- How it is used
- Uses streaming to prepare large training datasets and deliver LLM responses incrementally.
- How deep it goes
- Shows dataset loading and response-streaming examples, while noting that streaming services involve broader complexity.
- Examples
- Uses TinyStories training data and a simple chat application that displays generated responses incrementally.
- What you can do after
- Apply it via Hugging Face dataset loading and a client-side readable stream.
Effective Platform Engineering
An event stream is an ordered, continuously produced sequence of events that can be consumed by software systems and is not modified after creation.
- How it is used
- Coordinates team-related platform integrations and helps scale services as platform adoption grows.
- How deep it goes
- Explains consumers, asynchronous benefits, and configuration reconciliation, but does not specify stream implementation or delivery guarantees.
- Examples
- Uses team sync events to illustrate namespace configuration and integrations with Terraform Cloud, Jira, and observability tools.
- What you can do after
- Apply it by publishing platform events and assigning configuration responsibilities to independent consumers.
Read how Effective Platform Engineering explains stream processing →
Machine Learning Platform Engineering
Response streaming delivers generated output incrementally as it becomes available instead of waiting for the complete response.
- How it is used
- Deploys incremental responses in a Chainlit RAG chat interface so users see progress while an answer is generated.
- How deep it goes
- Shows an async generator yielding sources and model chunks, then integrates streaming into a validated chat handler.
- Examples
- Uses a documentation-answering RAG assistant with a Chainlit interface, source events, and token events.
- What you can do after
- Apply it via Chainlit by connecting an async RAG response generator to a validated chat handler.
Read how Machine Learning Platform Engineering explains stream processing →
Go in Practice, Second Edition
An event stream is an ordered flow of discrete event records delivered continuously over time.
- How it is used
- Processes large files, uploads, and log messages incrementally rather than waiting for all input.
- How deep it goes
- Works through line scanning, buffered copying, and multipart reading; discusses network reliability and response-buffering trade-offs.
- Examples
- Uses newline-separated JSON logs, file-to-file copying, multipart file uploads, network log messages, and template responses.
- What you can do after
- Build it from scratch with bufio.Scanner or MultipartReader, or apply it via io.Copy to move data incrementally.
Read how Go in Practice, Second Edition explains stream processing →
The Art of Code
Streaming is a processing technique that handles data incrementally as it becomes available instead of retaining the entire result in memory.
- How it is used
- Addresses export memory pressure while also using Java streams to express transformations declaratively.
- How deep it goes
- Traces batch fetching, disk writes, and memory release in an export, without specifying the export implementation.
- Examples
- Uses an Excel export and a Java pipeline that filters, uppercases, and collects words.
- What you can do after
- Evaluate/understand streaming and pipeline choices against memory pressure, readability, traversal costs, and parallel overhead.
Modern C, Third Edition
A stream is an abstract C input or output sequence associated with a file, device, or other data source or destination.
- How it is used
- Routes C library input and output through streams associated with files or standard destinations.
- How deep it goes
- Defines the opaque interface and demonstrates output and file modes, while leaving stream implementation details unexplored.
- Examples
- Uses character and string output, an append-mode log file, and formatted output directed to a stream.
- What you can do after
- Operate file streams by choosing access modes, checking whether opening succeeded, and directing output to a stream.
Read how Modern C, Third Edition explains stream processing →
Build Applications with Local AI Models on a Mac
Delivery of a generated response incrementally as its text becomes available, rather than all at once.
- How it is used
- Makes local chatbot responses feel responsive by displaying generated text as it arrives.
- How deep it goes
- Explains chunk iteration, immediate printing, and the flow between a Python script, Ollama server, and model.
- Examples
- Uses an internet-explanation prompt in a Python script and a Streamlit chat interface with an updating placeholder.
- What you can do after
- Apply it via Ollama’s Python library to display incremental model responses.
Read how Build Applications with Local AI Models on a Mac explains stream processing →
100 C++ Mistakes and How to Avoid Them
A stream is an interface that represents a sequential flow of data between a program and an input or output source.
- How it is used
- Replaces risky formatted C-style I/O and reduces housekeeping when processing container sequences.
- How deep it goes
- Explains stream operators, input failure checks, and transform-filter-sum operations without examining implementation internals.
- Examples
- Demonstrates parsing numeric and text values, and transforming, filtering, and summing integers.
- What you can do after
- Apply C++ stream operators for typed I/O and ranges and views for sequence transformations.
Read how 100 C++ Mistakes and How to Avoid Them explains stream processing →
Deep Learning with PyTorch, Second Edition
Sending generated output incrementally to a client as it becomes available rather than waiting for the full result to finish.
- How it is used
- Reduces response latency by delivering generated text incrementally to clients.
- How deep it goes
- Explains asynchronous token delivery alongside request queues, batching, and a background model worker.
- Examples
- Uses generated text tokens and a restaurant analogy of batched orders served as dishes become ready.
- What you can do after
- Build a FastAPI text-generation endpoint that streams generated tokens to clients.
Read how Deep Learning with PyTorch, Second Edition explains stream processing →
Digital Twins in Action
A data stream is a continuously or repeatedly produced sequence of data records from one or more sources.
- How it is used
- Keeps digital twins current by ingesting and processing continuously arriving data from physical systems.
- How deep it goes
- Explains sensor decoding, sliding-window updates, and video transmission choices, with illustrative processing code.
- Examples
- Uses temperature readings, live train-movement events, and webcam object detection to illustrate streams.
- What you can do after
- Evaluate stream-ingestion and video-transmission choices for a digital twin.
Read how Digital Twins in Action explains stream processing →
.NET in Action, Second Edition
Stream is a .NET abstraction for reading and writing sequences of bytes across sources such as files, network responses, memory, and console I/O.
- How it is used
- Uses streams to read and write data across files, network responses, memory, and console output.
- How deep it goes
- Demonstrates file and console stream setup, JSON parsing, and testing; defers append and seek use cases.
- Examples
- Uses JSON file processing, line-by-line text search, and a MemoryStream-backed unit test.
- What you can do after
- Apply .NET Stream APIs to process file or network data and supply in-memory test input.
Read how .NET in Action, Second Edition explains stream processing →
Bootstrapping Microservices, Second Edition: With Docker, Kubernetes, GitHub Actions, and Terraform
Node.js streaming is the incremental processing and transfer of data through Node.js stream objects rather than loading the complete data set at once.
- How it is used
- Delivers video through HTTP responses in a microservices application.
- How deep it goes
- Explains readable streams and piping, then stops short of stream internals.
- Examples
- Uses a local video file, Azure Storage, forwarding services, and a browser video element.
- What you can do after
- Build from scratch a basic Express endpoint that streams a local video file to an HTTP response.
Modern C
- How it is used
- Directs character and string output to terminal defaults or file-backed streams.
- How deep it goes
- Explains FILE* as opaque, demonstrates output and file opening, and defers pointer details and text-binary distinctions.
- Examples
- Demonstrates a manual puts implementation and writing a log file through fopen and fputs.
- What you can do after
- Apply fputs and fputc to output through a stream, or open a file and check whether opening succeeded.
ASP.NET Core in Action, 3e
File streaming is the technique of processing file content incrementally as it is transferred instead of storing the entire file before processing.
- How it is used
- Processes large request or upload data incrementally to avoid holding the entire file in memory at once.
- How deep it goes
- Defines request-body Stream and contrasts direct streaming with IFormFile buffering, but defers implementation details.
- Examples
- Contrasts buffered IFormFile uploads with direct streaming for large files.
- What you can do after
- Evaluate when to avoid IFormFile for large uploads and consider direct streaming instead.
Read how ASP.NET Core in Action, 3e explains stream processing →
Related concepts
- Apache Kafkaincludes · Apache Kafka is a distributed event-streaming platform for publishing, storing, and processing streams of records.
- promiseuses · A JavaScript Promise is an object representing the eventual completion or failure of an asynchronous operation and its resulting value or reason.
- pull modeluses · A pull model is a communication pattern in which a consumer initiates requests to retrieve data, messages, or work according to its needs or readiness.
- batchingcontrasts with · Batching is the grouping of multiple items, requests, or operations so they can be processed together as a unit.
- data streamuses · A data stream is an ordered sequence of data items that may be ongoing and is made available incrementally for consumption or processing.
- data aggregationuses · Data aggregation is the process of combining multiple data values, records, or observations into a consolidated or summarized result.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



















