liveBook · Manning

Stream processing

As explained in Designing AI Systems

A streaming response is an output delivery mode that sends generated content incrementally as it becomes available instead of waiting for the complete result.

A streaming response delivers generated content incrementally instead of waiting for the complete result. In a conversational interface, users can begin reading while generation continues; total generation time may stay the same, but the response feels faster. Streaming also permits early termination when a workflow detects problematic content. section 2.3.4 section 3.3

Progressive delivery requires every layer to forward chunks rather than buffer the complete response. A provider adapter normalizes provider-specific formats; the Model Service forwards chunks over gRPC; and the gateway converts the stream to Server-Sent Events (SSE) over HTTP for external clients. section 3.10 section 3.3.1

Listing 3.10 ChatChunk message for streaming responses
message ChatChunk {
  string token = 1;    #A
  string model = 2;    #B
  optional string finish_reason = 3;    #C
  optional TokenUsage usage = 4;
}

ChatChunk provides a provider-independent contract for each fragment and its completion metadata. section 3.3.2

A consumer should handle the stream as fragments, not complete words or complete responses:

  • Append each token fragment as it arrives; a fragment can be part of a word or punctuation. section 3.3.2
  • Use the model field to identify the producer. The final chunk can include a finish reason and token usage, so cost tracking generally happens after generation. section 3.3.2
  • Treat a final finish_reason of error as a mid-stream failure: retain the partial response and show an error indication. Fallback is usually restricted to failures before streaming starts because switching providers mid-response is disruptive. section 3.3.3

A Python workflow can call platform.models.chat_stream() and yield each chunk. With response_mode="stream", the runtime turns yielded values into SSE events; a final event can signal completion and include the session ID. The runtime can apply a per-chunk timeout to detect a stalled generator. section 3.9.3 section 9.1.4

Listing 9.5 A streaming workflow that yields tokens progressively
@workflow(
    name="patient_intake_assistant",
    api_path="/patient-assistant",
    response_mode="stream",    #A
    timeout_seconds=30
)
def handle_patient_question(question, patient_id):
    platform = GenAIPlatform()
    session = platform.sessions.get_or_create(
        patient_id
    )
 
    history, _ = platform.sessions.get_messages(
        session.session_id
    )
    stream = platform.models.chat_stream(    #B
        model="gpt-4o",    #B
        system_prompt_name="patient_intake_assistant",
        messages=[*history,
                  {"role": "user",
                   "content": question}]
    )
 
    for chunk in stream:    #C
        yield {"token": chunk.token}    #C
 
    yield {"done": True,
           "session_id": session.session_id}

This workflow streams a patient-assistant response and yields a separate completion event. section 9.1.4

Streaming an agent response

An agent can open a model stream on each loop iteration, yield content tokens, and buffer tool-call deltas until the turn ends. An incremental output filter can hold short spans before release to detect violations, trading some time-to-first-token for protection against exposing unfiltered content. section 10.9.1

How Designing AI Systems covers it

How deep it goes
Traces chunks across providers, adapters, gRPC, gateway, and SSE, then implements workflow streaming and mid-stream error handling.
Examples
Uses a patient-assistant workflow and a streaming agent loop to illustrate progressive delivery and buffered tool calls.
What you can do after
Builds a Python streaming workflow that yields model chunks and signals completion to an SSE client.
How it connects
Connects streaming responses to provider adapters, gRPC, Server-Sent Events, fallback, tool calls, and output guardrails.

Also explained in

Compare how 25 books explain stream processing →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.