liveBook · Manning

Response mode

As explained in Designing AI Systems

A response mode is a workflow configuration that determines whether results are returned synchronously or incrementally as a stream.

A response mode configures how a workflow runtime turns a function’s output into an HTTP response. The runtime reads response_mode from workflow metadata and selects a handler; the choice determines whether clients wait for a result, receive it progressively, or retrieve it as a background job. section 9.1.2

Choose the mode for the interaction and expected duration:

  • Synchronous (sync): execute inline, wait for the function to return, and send one JSON response. This suits fast operations, such as those taking less than five seconds. section 9.1.2
  • Streaming (stream): keep the connection open and send each yielded value as a Server-Sent Event (SSE). This suits conversational generation where clients benefit from progressive output. section 9.1.2 section 9.1.4
  • Asynchronous (async): create a job, return HTTP 202 with a job ID and status URL, and run the function in the background. This suits long-running research or batch work. section 9.1.2 section 9.1.5

A streaming workflow uses yield rather than return. Its handler validates the request, propagates trace context, and applies a timeout to each request for the next chunk. A completion marker detects generator exhaustion without allowing StopIteration to surface as a RuntimeError inside the task. section 9.1.4

Listing 9.5 A streaming workflow that yields tokens progressively
@workflow(
    name="patient_intake_assistant",
    api_path="/patient-assistant",
    response_mode="stream",    #A
    timeout_seconds=30
)
def handle_patient_question(question, patient_id):
    platform = GenAIPlatform()
    session = platform.sessions.get_or_create(
        patient_id
    )
 
    history, _ = platform.sessions.get_messages(
        session.session_id
    )
    stream = platform.models.chat_stream(    #B
        model="gpt-4o",    #B
        system_prompt_name="patient_intake_assistant",
        messages=[*history,
                  {"role": "user",
                   "content": question}]
    )
 
    for chunk in stream:    #C
        yield {"token": chunk.token}    #C
 
    yield {"done": True,
           "session_id": session.session_id}

A patient assistant configures streaming, yields model tokens, and finishes with a completion signal and session ID. section 9.1.4

An asynchronous workflow can still be an ordinary function that returns a dictionary. The handler records the original input for crash recovery, schedules background execution, and returns the job reference. A job ID in context also lets the function report progress without receiving that ID directly. section 9.1.5

Workflow-to-workflow calls can hide these differences from the parent: platform.workflows.call() returns synchronous JSON directly, polls an asynchronous job until completion, or consumes a stream into a list. The last approach means the parent receives the collected stream rather than progressive chunks. section 9.4.3

Listing 9.18 Routing a workflow call based on the child’s response mode
def call(self, api_path, body):
    response = self._http_post(    #A
        api_path, body
    )
 
    content_type = response.headers.get(
        "content-type", ""
    )
 
    if response.status_code == 200:    #B
        if "text/event-stream" in content_type:
            return self._consume_stream(
                response
            )
        return response.json()
 
    if response.status_code == 202:    #C
        job_id = response.json()["job_id"]
        return self._poll_until_complete(
            job_id
        )
    
    logger.error(    #D
        "child workflow %s returned %s: %s",    #D
        api_path, response.status_code,    #D
        response.text    #D
    )    #D
    raise WorkflowCallError(    #D
        f"Child workflow {api_path} failed"
        f" with status {response.status_code}"
    )

The call dispatcher distinguishes JSON, event streams, and accepted jobs using the HTTP status and content type. section 9.4.3

Parallel research

An async research workflow fans out to knowledge-base and web-search children, then synthesizes their results. The client polls the job instead of keeping a connection open; independently scaled child containers keep research bursts from starving the conversational workflow. section 10.12

How Designing AI Systems covers it

How deep it goes
Explains handler selection, streaming timeouts, background job creation, and client-side response dispatch.
Examples
Uses a patient assistant that streams tokens and a research workflow that runs parallel searches asynchronously.
What you can do after
Apply it via the workflow decorator to choose a response pattern and handle its HTTP result.
How it connects
Connects response modes to Server-Sent Events, background jobs, polling, timeouts, and workflow composition.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.