Asynchronous programming lets independent tasks proceed without waiting for one another, then delivers each result as it becomes available. For agent systems, this reduces the elapsed time of parallel LLM requests from the sum of their durations to roughly the longest request. section 2.2.6
Python provides the core execution model:
- Define coroutines with async def. section 2.2.6
- Use await to suspend the current coroutine while an asynchronous operation completes, allowing other tasks to run. section 2.2.6
- Use asyncio.gather to start multiple coroutines concurrently and collect their results. section 2.2.6
- Call LiteLLM through acompletion when the concurrent operation is an LLM request. section 2.2 section 2.2.6
Parallel LLM questions
Create one coroutine for each prompt, gather the responses, and pair each answer with its question. Three one-second requests can complete in about one second instead of three when run concurrently. section 2.2.6
Production-oriented concurrency adds two safeguards:
- Wrap the acompletion call inside asyncio.Semaphore to cap active API calls, even when many tasks have been created. section 2.2.6
- Set num_retries to retry transient failures with exponential backoff. section 2.2.6
Asynchronous workflows can also pause and resume at approval boundaries. Persisting pending tool calls and their approval decisions lets an agent wait for human confirmation before continuing. section 6.3.5
