A streaming response delivers generated content incrementally instead of waiting for the complete result. In a conversational interface, users can begin reading while generation continues; total generation time may stay the same, but the response feels faster. Streaming also permits early termination when a workflow detects problematic content. section 2.3.4 section 3.3
Progressive delivery requires every layer to forward chunks rather than buffer the complete response. A provider adapter normalizes provider-specific formats; the Model Service forwards chunks over gRPC; and the gateway converts the stream to Server-Sent Events (SSE) over HTTP for external clients. section 3.10 section 3.3.1
Listing 3.10 ChatChunk message for streaming responses
message ChatChunk {
string token = 1; #A
string model = 2; #B
optional string finish_reason = 3; #C
optional TokenUsage usage = 4;
}ChatChunk provides a provider-independent contract for each fragment and its completion metadata. section 3.3.2
A consumer should handle the stream as fragments, not complete words or complete responses:
- Append each token fragment as it arrives; a fragment can be part of a word or punctuation. section 3.3.2
- Use the model field to identify the producer. The final chunk can include a finish reason and token usage, so cost tracking generally happens after generation. section 3.3.2
- Treat a final finish_reason of error as a mid-stream failure: retain the partial response and show an error indication. Fallback is usually restricted to failures before streaming starts because switching providers mid-response is disruptive. section 3.3.3
A Python workflow can call platform.models.chat_stream() and yield each chunk. With response_mode="stream", the runtime turns yielded values into SSE events; a final event can signal completion and include the session ID. The runtime can apply a per-chunk timeout to detect a stalled generator. section 3.9.3 section 9.1.4
Listing 9.5 A streaming workflow that yields tokens progressively
@workflow(
name="patient_intake_assistant",
api_path="/patient-assistant",
response_mode="stream", #A
timeout_seconds=30
)
def handle_patient_question(question, patient_id):
platform = GenAIPlatform()
session = platform.sessions.get_or_create(
patient_id
)
history, _ = platform.sessions.get_messages(
session.session_id
)
stream = platform.models.chat_stream( #B
model="gpt-4o", #B
system_prompt_name="patient_intake_assistant",
messages=[*history,
{"role": "user",
"content": question}]
)
for chunk in stream: #C
yield {"token": chunk.token} #C
yield {"done": True,
"session_id": session.session_id}This workflow streams a patient-assistant response and yields a separate completion event. section 9.1.4
Streaming an agent response
An agent can open a model stream on each loop iteration, yield content tokens, and buffer tool-call deltas until the turn ends. An incremental output filter can hold short spans before release to detect violations, trading some time-to-first-token for protection against exposing unfiltered content. section 10.9.1
























