liveBook · Manning

Concurrency

As explained in Designing AI Systems

Concurrency is the management of multiple in-progress operations so that their execution can overlap without requiring simultaneous CPU execution.

Concurrency manages multiple in-progress operations by overlapping execution, even when one process is waiting on platform services. An asyncio event loop multiplexes requests, while multiple worker processes increase capacity inside a container. section 9.3.1

A production workflow service combines several concurrency strategies:

  • Use an event loop to accept and begin requests while earlier requests wait for gRPC responses. section 9.3.1
  • Run multiple uvicorn worker processes when one process reaches its limits; each process owns an event loop and thread pool. section 9.3.1
  • Dispatch synchronous workflows to threads so blocking calls do not prevent other requests from progressing. section 9.3.1
  • Fan out independent child workflows with `call_parallel()`, then collect results in input order before continuing. section 9.4.2

Parallel research calls

A research workflow sends independent paper, news, and patent searches through `platform.workflows.call_parallel()`. The SDK waits for all results, then passes them to a synthesis model. Each child call retains the normal retry and response-mode behavior; a failure after retries propagates to the parent. section 9.4.2

Listing 9.12 Runtime server entry point with worker configuration
app = build_app(    #A
    os.environ["WORKFLOW_MODULE"]    #A
)    #A

if __name__ == "__main__":
    uvicorn.run(
        "genai_platform.runtime.server:app",    #B
        host="0.0.0.0",
        port=8000,
        workers=4    #C
    )

Shows a runtime entry point configuring four uvicorn workers for parallel request handling. section 9.3.1

Listing 9.17 Executing multiple workflow calls concurrently
from concurrent.futures import (
    ThreadPoolExecutor
)
 
def call_parallel(self, calls):    #A
    with ThreadPoolExecutor(
        max_workers=len(calls)
    ) as pool:
        futures = [
            pool.submit(    #B
                self.call,
                api_path,
                body
            )
            for api_path, body in calls
        ]
 
        results = []
        for future in futures:    #C
            results.append(
                future.result()
            )
        return results

Shows `ThreadPoolExecutor` submitting workflow calls concurrently and collecting futures in the original order. section 9.4.2

How Designing AI Systems covers it

How deep it goes
Explains event loops, worker processes, thread pools, fan-out, result ordering, and failure propagation through implementation examples.
Examples
Uses uvicorn workers, synchronous workflow threads, and a research assistant that queries papers, news, and patents in parallel.
What it assumes
Assumes workflow services, HTTP gateways, gRPC calls, and synchronous or asynchronous workflow modes.
What you can do after
Apply concurrency via asyncio, uvicorn workers, and `workflow.call_parallel()` to increase throughput and reduce independent-call latency.
How it connects
Connects asyncio event loops, worker processes, thread pools, gRPC, workflow composition, retries, and HTTP gateways.

Also explained in

Compare how 31 books explain concurrency →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.