liveBook · Manning

Context variable

As explained in Designing AI Systems

A context variable is a dynamically scoped variable whose value can differ across execution contexts such as threads or asynchronous tasks.

A context variable carries a value within an execution context without requiring every function to accept it as an argument. In a workflow runtime, it can carry a trace context or job ID while keeping concurrent requests isolated. section 9.3.1 section 9.1.3

For a synchronous workflow, the handler extracts trace context from request headers and sets it before invoking the developer’s function. The GenAIPlatform constructor reads the value, so subsequent service calls are linked to the same distributed trace without adding tracing parameters to the workflow signature. section 9.1.3

Listing 9.4 Synchronous request handler
import asyncio
import inspect
from fastapi import Request
from fastapi.responses import JSONResponse
 
def create_sync_handler(workflow_func, timeout):
    sig = inspect.signature(workflow_func)    #C
 
    async def handle(request: Request):
        body = await request.json()
        trace_ctx = extract_trace_context(    #A
            request.headers    #A
        )    #A
        _trace_var.set(trace_ctx)    #B
 
        try:
            sig.bind(**body)    #C
        except TypeError as e:    #C
            return JSONResponse(    #C
                {"error": str(e)},    #C
                status_code=422    #C
            )    #C
 
        try:    #D
            result = await asyncio.wait_for(    #D
                asyncio.to_thread(    #D
                    workflow_func, **body    #D
                ),    #D
                timeout=timeout    #D
            )    #D
        except asyncio.TimeoutError:    #E
            return JSONResponse(    #E
                {"error": "Request timed out"},    #E
                status_code=504    #E
            )    #E
        except Exception as e:    #F
            logger.exception(    #F
                "workflow request failed")    #F
            return JSONResponse(    #F
                {"error": "An internal error"    #F
                          " occurred. Please"    #F
                          " try again."},    #F
                status_code=500    #F
            )    #F
 
        return JSONResponse(result)    #G
    return handle

The handler sets the trace context before dispatching the workflow to a thread. section 9.1.3

Python’s contextvars are per asyncio task by default. When the synchronous handler uses asyncio.to_thread, the thread receives a copy of the current context. Simultaneous requests can therefore carry different trace contexts and job IDs without cross-contamination. section 9.3.1

Progress and checkpoints

An async handler sets the job ID before starting a background task. A workflow can then call update_job_progress() with only a message; the client reads the job ID from context and updates the job record. Checkpoint operations use the same pattern, while the saved checkpoint itself is stored in the job record for recovery after a crash. section 9.2.2

How Designing AI Systems covers it

How deep it goes
Explains context propagation and isolation across asyncio tasks and threads, with a handler setting trace context.
Examples
Uses a synchronous request handler and an asynchronous research workflow with progress updates and checkpoints.
What you can do after
Evaluate how context variables keep concurrent workflow requests isolated while carrying trace and job identifiers.
How it connects
Connects context variables to distributed tracing, asynchronous tasks, threads, and checkpointing.

Also explained in

Compare how 3 books explain context variable →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.