liveBook · Manning

Workflow runtime server

As explained in Designing AI Systems

A workflow runtime server is SDK library code that imports a workflow module, registers an HTTP route, receives requests, invokes a workflow function, and returns its responses as a service.

A workflow runtime server turns a decorated Python function into an HTTP service without requiring the function to handle HTTP. It is SDK library code, not a separate service: the deployment pipeline sets it as the entry point of a container holding the workflow code and its dependencies. section 9.1 section 9.1.1

The server is part of the data plane. Deployment, scaling, and health monitoring belong to the Workflow Service control plane. Synchronous and streaming requests do not pass through that control plane; asynchronous requests use it to manage job state. section 9.8 section 9.0

At startup, the server imports the developer’s module and selects a decorated function by WORKFLOW_NAME. It reads the function’s metadata, chooses a handler for its response mode, and registers its api_path with FastAPI. JSON request fields become keyword arguments; return values become responses. Uvicorn listens for HTTP traffic, while calls to platform services use internal gRPC. section 9.1.2 section 9.8

Listing 9.3 The workflow runtime server
import uvicorn
import os
import importlib
import inspect
from fastapi import FastAPI
 
def find_workflow(module_name):    #A
    module = importlib.import_module(module_name)
    target = os.environ["WORKFLOW_NAME"]    #B
    for name, obj in inspect.getmembers(module):
        if hasattr(obj, '_workflow_metadata'):
            meta = obj._workflow_metadata
            if meta['name'] == target:
                return obj, meta
    raise ValueError(
        f"No workflow named '{target}' found"
    )
 
def build_app(module_name):
    func, meta = find_workflow(module_name)
    mode = meta['response_mode']    #C
    timeout = meta['reliability'][    #D
        'timeout_seconds'    #D
    ]    #D
 
    if mode == "sync":    #E
        handler = create_sync_handler(    #E
            func, timeout    #E
        )    #E
    elif mode == "stream":    #E
        handler = create_stream_handler(    #E
            func, timeout    #E
        )    #E
    elif mode == "async":    #E
        handler = create_async_handler(func)    #E
 
    app = FastAPI()    #F
    app.add_api_route(    #F
        meta['api_path'], handler,    #F
        methods=["POST"]    #F
    )    #F
 
    @app.get("/health/live")    #G
    def live():    #G
        return {"status": "ok"}    #G
 
    @app.get("/health/ready")    #G
    def ready():    #G
        return {"status": "ok"}    #G
 
    return app

The runtime server listing connects module discovery, metadata, handler selection, and route registration. section 9.1.2

The response mode determines how the handler exposes workflow execution:

  • Synchronous: run with a configurable timeout and return one JSON response. section 9.8
  • Streaming: iterate over a generator and send Server-Sent Events progressively. section 9.8
  • Asynchronous: create a job record, return HTTP 202 with a job ID and status URL, and run the function in a background thread. section 9.1.5

For an asynchronous job, the runtime server reports progress and records completion or failure with the Workflow Service. Clients poll the service’s job endpoint rather than the workflow container. The job record also retains the original request and assigned container so recovery logic can identify and re-dispatch orphaned work. section 9.5 section 9.2.1

Listing 9.8 Asynchronous request handler
import asyncio
 
def create_async_handler(workflow_func):
    async def handle(request: Request):
        body = await request.json()
        trace_ctx = extract_trace_context(    #A
            request.headers    #A
        )    #A
        _trace_var.set(trace_ctx)    #B
        _job_id_var.set(None)    
        platform = GenAIPlatform()
 
        job = platform.workflows.create_job(    #C
            workflow_id=workflow_func
                ._workflow_metadata['name'],
            input_json=json.dumps(body)    #C
        )
        _job_id_var.set(job.job_id)    #D
 
        asyncio.create_task(    #E
            run_async_workflow(
                platform, workflow_func,
                body, job.job_id
            )
        )
 
        return JSONResponse(    #F
            {    #F
                "job_id": job.job_id,    #F
                "status": "processing",    #F
                "status_url":    #F
                    f"/jobs/{job.job_id}"    #F
            },    #F
            status_code=202    #F
        )    #F
    return handle
 
async def run_async_workflow(
    platform, func, body, job_id    #G
):
    try:
        result = await asyncio.to_thread(    #H
            func, **body    #H
        )    #H
        platform.workflows.complete_job(    #I
            job_id, json.dumps(result)    #I
        )    #I
    except Exception as e:
        platform.workflows.fail_job(    #J
            job_id, str(e)    #J
        )    #J

The asynchronous handler shows job creation, context setup, background scheduling, and the 202 response. section 9.1.5

Decorator metadata separates operational concerns: scaling and resource settings flow through deployment, while reliability settings are enforced inside the container. The server also propagates trace context to SDK service calls. Multiple Uvicorn workers and per-thread context copies isolate concurrent requests. section 9.8

How Designing AI Systems covers it

How deep it goes
Implements workflow discovery, routing, response modes, asynchronous job handling, tracing, and concurrency isolation.
Examples
Uses a patient-intake workflow and a long-running research workflow to illustrate decorated functions and asynchronous job handling.
What you can do after
Build it from scratch using the illustrated Python runtime, HTTP handlers, and asynchronous job lifecycle.
How it connects
Connects workflow decorators, HTTP routing, gRPC, asynchronous jobs, distributed tracing, and control-plane separation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.