liveBook · Manning

Asyncio.to_thread

As explained in Designing AI Systems

asyncio.to_thread is a Python API that runs a synchronous callable in a separate thread while allowing the calling coroutine to remain asynchronous.

asyncio.to_thread runs a synchronous workflow function in a worker thread, allowing the asyncio event loop to accept and coordinate other HTTP requests while that function waits on downstream services. section 9.1.3 section 9.3.1

A synchronous request handler combines asyncio.to_thread with asyncio.wait_for. The timeout applies to the complete workflow, returns 504 when the budget expires, and aligns downstream gRPC deadlines with the remaining wall-clock budget. Separate worker processes provide separate event loops and thread pools; concurrent requests retain isolated service clients and context state. section 9.1.3 section 9.3.1

Listing 9.4 Synchronous request handler
import asyncio
import inspect
from fastapi import Request
from fastapi.responses import JSONResponse
 
def create_sync_handler(workflow_func, timeout):
    sig = inspect.signature(workflow_func)    #C
 
    async def handle(request: Request):
        body = await request.json()
        trace_ctx = extract_trace_context(    #A
            request.headers    #A
        )    #A
        _trace_var.set(trace_ctx)    #B
 
        try:
            sig.bind(**body)    #C
        except TypeError as e:    #C
            return JSONResponse(    #C
                {"error": str(e)},    #C
                status_code=422    #C
            )    #C
 
        try:    #D
            result = await asyncio.wait_for(    #D
                asyncio.to_thread(    #D
                    workflow_func, **body    #D
                ),    #D
                timeout=timeout    #D
            )    #D
        except asyncio.TimeoutError:    #E
            return JSONResponse(    #E
                {"error": "Request timed out"},    #E
                status_code=504    #E
            )    #E
        except Exception as e:    #F
            logger.exception(    #F
                "workflow request failed")    #F
            return JSONResponse(    #F
                {"error": "An internal error"    #F
                          " occurred. Please"    #F
                          " try again."},    #F
                status_code=500    #F
            )    #F
 
        return JSONResponse(result)    #G
    return handle

The request-handler listing shows the operational sequence: parse JSON, extract tracing context, validate arguments, dispatch the synchronous callable with asyncio.to_thread, and enforce a timeout. section 9.1.3

Synchronous workflow request

A POST request containing a patient question and identifier remains open while the workflow executes. A successful response returns structured JSON with the generated response and sources; longer-running workflows may require streaming or asynchronous mode. section 9.1.3

How Designing AI Systems covers it

How deep it goes
Explains thread dispatch, timeout composition, worker pools, and request isolation, but does not detail thread cancellation or scheduling internals.
Examples
Demonstrates a FastAPI-style request handler and a patient-assistant HTTP exchange returning structured JSON.
What you can do after
Operate synchronous workflows through a nonblocking request handler with coordinated timeout and concurrency behavior.
How it connects
Connects synchronous workflows with asyncio event loops, thread pools, HTTP timeouts, gRPC deadlines, and multi-process worker configuration.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.