liveBook · Manning

Uvicorn

As explained in Designing AI Systems

Uvicorn is an ASGI server that listens for network connections and dispatches incoming requests to an ASGI application.

Uvicorn is an ASGI server that listens on a port and dispatches incoming HTTP requests to an ASGI application. With FastAPI, it provides the external HTTP boundary for workflow services while internal service calls use gRPC. section 9.1.2

Its asyncio event loop multiplexes requests while they wait on platform-service responses. Production deployments add worker processes, each with an independent event loop and thread pool, to increase parallelism within a container. section 9.3.1

A runtime server can select response handling by workflow mode:

  • Synchronous handlers return one JSON object after inline execution. section 9.1.2
  • Streaming handlers yield progressive output through Server-Sent Events. section 9.1.2
  • Asynchronous handlers start background work and return a job identifier with HTTP 202. section 9.1.2
  • Worker counts should reflect CPU allocation and workflows that spend substantial time waiting on gRPC responses. section 9.3.1
Listing 9.3 The workflow runtime server
import uvicorn
import os
import importlib
import inspect
from fastapi import FastAPI
 
def find_workflow(module_name):    #A
    module = importlib.import_module(module_name)
    target = os.environ["WORKFLOW_NAME"]    #B
    for name, obj in inspect.getmembers(module):
        if hasattr(obj, '_workflow_metadata'):
            meta = obj._workflow_metadata
            if meta['name'] == target:
                return obj, meta
    raise ValueError(
        f"No workflow named '{target}' found"
    )
 
def build_app(module_name):
    func, meta = find_workflow(module_name)
    mode = meta['response_mode']    #C
    timeout = meta['reliability'][    #D
        'timeout_seconds'    #D
    ]    #D
 
    if mode == "sync":    #E
        handler = create_sync_handler(    #E
            func, timeout    #E
        )    #E
    elif mode == "stream":    #E
        handler = create_stream_handler(    #E
            func, timeout    #E
        )    #E
    elif mode == "async":    #E
        handler = create_async_handler(func)    #E
 
    app = FastAPI()    #F
    app.add_api_route(    #F
        meta['api_path'], handler,    #F
        methods=["POST"]    #F
    )    #F
 
    @app.get("/health/live")    #G
    def live():    #G
        return {"status": "ok"}    #G
 
    @app.get("/health/ready")    #G
    def ready():    #G
        return {"status": "ok"}    #G
 
    return app

Shows a runtime-server construction that discovers a workflow, reads response metadata, and selects a handler. section 9.1.2

Listing 9.12 Runtime server entry point with worker configuration
app = build_app(    #A
    os.environ["WORKFLOW_MODULE"]    #A
)    #A

if __name__ == "__main__":
    uvicorn.run(
        "genai_platform.runtime.server:app",    #B
        host="0.0.0.0",
        port=8000,
        workers=4    #C
    )

Shows Uvicorn starting an application on all interfaces, port 8000, with four worker processes. section 9.3.1

How Designing AI Systems covers it

How deep it goes
Explains event-loop multiplexing, thread dispatch, response modes, and worker processes, but does not detail Uvicorn internals.
Examples
Uses a workflow runtime server and a four-worker entry point listening on port 8000.
What it assumes
Assumes familiarity with Python services, FastAPI, asyncio, and workflow response modes.
What you can do after
Operate/configure Uvicorn to serve an ASGI workflow application with selected response handling and multiple workers.
How it connects
Connects Uvicorn to ASGI, FastAPI, asyncio, gRPC, Server-Sent Events, and worker processes.

Also explained in

Compare how 2 books explain Uvicorn →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.