liveBook · Manning

Stream processing

As explained in Enterprise RAG

Streaming is the incremental delivery of a language model's output as partial results become available.

Streaming delivers a language model’s answer incrementally instead of making the user wait for the complete response. In retrieval workflows, it is most valuable during final answer generation, which can take substantially longer than search. Starting word-by-word output after initial processing reduces blank-screen time and improves perceived responsiveness. section 4.4

A production-oriented streaming workflow separates retrieval from answer delivery:

  • Search agents retrieve relevant documents and return structured results before writing begins. section 4.1.2
  • A writer agent receives the retrieved information and streams the final answer to the frontend. section 4.4
  • Enable streaming in the model configuration with a stream setting, while constraining the writer to retrieved facts. section 4.4

Streaming a retrieved answer

A group chat routes a user question through search agents. After sufficient product information is retrieved, the writer assistant generates a concise response with temperature set to zero and streaming enabled, returning partial text one word at a time. section 4.4 section 4.1.2

Figure 4.11 After the 2 agents in the groupchat retrieve enough information to answer the user’s question, the structured search results flow to a Writer Agent, which streams back a concise answer to the user, one word at a time.

Follow the retrieval-to-writer flow showing structured search results becoming an incrementally delivered answer. section 4.4

How Enterprise RAG covers it

How deep it goes
Explains the retrieval-to-writer workflow and configuration switch, but stops short of frontend transport or buffering details.
Examples
Uses a product-question workflow where a writer agent streams a concise answer word by word after document retrieval.
What it assumes
Assumes familiarity with language-model agents, retrieval, frontend delivery, and model configuration.
What you can do after
Apply it via AutoGen by enabling streaming on the writer agent and presenting partial answers to the frontend.
How it connects
Connects language models with retrieval-augmented generation, AutoGen agents, group chat, and Azure AI Search.

Also explained in

Compare how 25 books explain stream processing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Enterprise RAG. The section links point to the parts of the book it draws on.