LiteLLM is a Python library for calling LLMs from different providers through one interface. In an agent, this makes it easier to compare models or switch when cost, performance, or availability changes without rewriting each provider call. Its completion function accepts a model name and messages, returns a response in the OpenAI format, and reads provider API keys from environment variables. section 2.2.3 section 2.1.3
Listing 2.4 Unified API calls with LiteLLM
from litellm import completion
response = completion(
model="gpt-5-mini",
messages=[{"role": "user", "content": "Hello!"}]
)
response = completion(
model="claude-sonnet-4-5",
messages=[{"role": "user", "content": "Hello!"}]
)
Compare the two calls: the messages and function stay the same while the model name changes from GPT-5 mini to Claude Sonnet 4.5. section 2.2.3
For concurrent agent work, use acompletion with Python’s async/await and asyncio.gather. This lets requests run together rather than waiting for each one in sequence. Concurrency also needs limits: place the acompletion call inside an asyncio.Semaphore block to cap simultaneous API calls, and set num_retries for automatic retries with exponential backoff after transient failures. section 2.2.6
Listing 2.9 Rate-limited concurrent requests
import asyncio
from litellm import acompletion
# Limit to 10 concurrent requests
semaphore = asyncio.Semaphore(10)
async def call_llm(prompt: str) -> str:
"""LLM call with rate limiting and automatic retry."""
async with semaphore:
response = await acompletion(
model="gpt-5-mini",
messages=[{"role": "user", "content": prompt}],
num_retries=3 # Automatic retry with exponential backoff
)
return response.choices[0].message.content
# Even with 100 concurrent tasks, only 10 API calls run at a time
prompts = [f"What is {i} + {i}?" for i in range(100)]
tasks = [call_llm(p) for p in prompts]
results = await asyncio.gather(*tasks, return_exceptions=True)
The wrapper caps 100 scheduled tasks at 10 simultaneous API calls and gives each call up to three automatic retries. gather returns exceptions alongside results for the caller to handle. section 2.2.6
