liveBook · Manning

Server-sent events

As explained in Designing AI Systems

Server-Sent Events is an HTTP-based mechanism for continuously delivering unidirectional, incrementally generated events from a server to a client.

Server-Sent Events (SSE) deliver a stream of events from a server to a client over HTTP. For AI responses, SSE lets a conversational interface display text as it is generated instead of waiting for the complete answer. It differs from a synchronous response returned all at once and from an asynchronous job whose status is polled later. section 2.3.4 section 9.1.4

A client can request a stream with Accept: text/event-stream. The response uses Content-Type: text/event-stream and sends successive data fields, each containing a JSON value. A final event can carry a completion flag and session ID so the client knows the workflow has finished. section 2.3.4 section 9.1.4

HTTP/1.1 200 OK
Content-Type: text/event-stream
 
data: {"token": "Please"}
 
data: {"token": " bring"}
 
data: {"token": " your"}
 
data: {"token": " insurance"}
 
data: {"token": " card"}
 
data: {"done": true, "session_id": "sess-456"}

Inspect the response headers, token events, and final completion event. section 9.1.4

A streaming workflow calls platform.models.chat_stream(), iterates over the returned chunks, and yields each token. Setting response_mode="stream" selects a runtime handler that turns yielded values into SSE events. section 9.1.4

Listing 9.5 A streaming workflow that yields tokens progressively
@workflow(
    name="patient_intake_assistant",
    api_path="/patient-assistant",
    response_mode="stream",    #A
    timeout_seconds=30
)
def handle_patient_question(question, patient_id):
    platform = GenAIPlatform()
    session = platform.sessions.get_or_create(
        patient_id
    )
 
    history, _ = platform.sessions.get_messages(
        session.session_id
    )
    stream = platform.models.chat_stream(    #B
        model="gpt-4o",    #B
        system_prompt_name="patient_intake_assistant",
        messages=[*history,
                  {"role": "user",
                   "content": question}]
    )
 
    for chunk in stream:    #C
        yield {"token": chunk.token}    #C
 
    yield {"done": True,
           "session_id": session.session_id}

See the workflow configuration, token yields, and completion signal together. section 9.1.4

The runtime validates request arguments and propagates trace context before iterating over the workflow generator. A per-chunk timeout detects a stalled model stream; a sentinel marks generator exhaustion without letting StopIteration escape from the task. section 9.1.4

Listing 9.6 The runtime server's SSE handler for streaming workflows
from fastapi.responses import StreamingResponse
 
_DONE = object()    #A
 
def create_stream_handler(workflow_func, timeout):
    sig = inspect.signature(workflow_func)
 
    async def handle(request: Request):
        body = await request.json()
        trace_ctx = extract_trace_context(    #B
            request.headers    #B
        )    #B
        _trace_var.set(trace_ctx)    #C
 
        try:
            sig.bind(**body)    #D
        except TypeError as e:    #D
            return JSONResponse(    #D
                {"error": str(e)},    #D
                status_code=422    #D
            )    #D
 
        async def event_stream():
            gen = workflow_func(**body)    #E
            try:    #F
                while True:
                    chunk = await asyncio.wait_for(    #G
                        asyncio.to_thread(    #G
                            next, gen, _DONE    #G
                        ),    #G
                        timeout=timeout    #G
                    )    #G
                    if chunk is _DONE:    #H
                        break    #H
                    data = json.dumps(chunk)
                    yield f"data: {data}\n\n"
            except asyncio.TimeoutError:    #I
                yield f"data: {json.dumps({'error': 'Stream timed out'})}\n\n"
            except Exception:    #J
                logger.exception("stream failed")    #J
                yield f"data: {json.dumps({'error': 'Stream interrupted'})}\n\n"
 
        return StreamingResponse(    #K
            event_stream(),    #K
            media_type="text/event-stream"    #K
        )    #K
    return handle

Inspect the handler's request validation and generator iteration. section 9.1.4

SSE can also sit at both ends of a larger streaming pipeline. Provider adapters normalize differing provider stream formats; the Model Service forwards chunks through gRPC streaming; and the API gateway converts that stream to SSE for external clients. Following these boundaries helps locate streaming latency. section 3.3.1

How Designing AI Systems covers it

How deep it goes
Explains event formatting and a workflow handler, including validation, generator exhaustion, and per-chunk timeouts.
Examples
Uses a patient intake assistant that yields tokens and a final session ID.
What you can do after
Operate a Python streaming workflow by configuring response mode, yielding chunks, and sending a completion event.
How it connects
Connects HTTP streaming with gRPC streaming, provider adapters, API gateways, and asynchronous job polling.

Also explained in

Compare how 4 books explain server-sent events →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.