A rate limit is a provider-imposed bound on API use. It matters when an agent sends many LLM requests concurrently: asynchronous calls can shorten a batch’s elapsed time, but an unrestricted burst can trigger rate-limit errors. Network failures and server overload are separate reasons requests may fail. section 2.2.6
Use an asyncio.Semaphore to bound simultaneous API calls. Put the awaited LiteLLM acompletion call inside an async with block: tasks beyond the semaphore’s capacity wait, and a waiting task proceeds when a call finishes. This controls concurrency, not every possible provider quota over a time period. section 2.2.6
A hundred tasks, ten calls at a time
Create asyncio.Semaphore(10), then schedule 100 call_llm tasks with asyncio.gather. Only ten tasks can be inside the protected API-call block at once. Set num_retries=3 on acompletion for automatic retries with exponential backoff; use return_exceptions=True with gather so one failed task does not discard successful results. Retry handling complements, rather than replaces, the concurrency bound. section 2.2.6
Listing 2.9 Rate-limited concurrent requests
import asyncio
from litellm import acompletion
# Limit to 10 concurrent requests
semaphore = asyncio.Semaphore(10)
async def call_llm(prompt: str) -> str:
"""LLM call with rate limiting and automatic retry."""
async with semaphore:
response = await acompletion(
model="gpt-5-mini",
messages=[{"role": "user", "content": prompt}],
num_retries=3 # Automatic retry with exponential backoff
)
return response.choices[0].message.content
# Even with 100 concurrent tasks, only 10 API calls run at a time
prompts = [f"What is {i} + {i}?" for i in range(100)]
tasks = [call_llm(p) for p in prompts]
results = await asyncio.gather(*tasks, return_exceptions=True)
The complete concurrent-request listing shows where to acquire the semaphore, place the API call, configure retries, and collect failures. section 2.2.6
When a batch uses multiple providers, give each provider its own semaphore. A shared semaphore could let a burst for one provider block work destined for another. Example starting values are 30 concurrent OpenAI calls and 10 Anthropic calls; these are not universal limits. Check the provider’s documentation or dashboard, lower the value after rate-limit errors, and raise it only if the account’s limits permit greater throughput. section 2.4.2
Listing 2.12 Provider-specific rate limiting
PROVIDER_SEMAPHORES = {
"openai": asyncio.Semaphore(30),
"anthropic": asyncio.Semaphore(10),
}
def get_provider(model: str) -> str:
"""Extract provider name from model string."""
return "anthropic" if model.startswith("anthropic/") else "openai"
The provider-to-semaphore mapping shows how a model name selects an independent concurrency budget. section 2.4.2
The same concern applies to agent tools that call external services. Slack, Google Drive, and Salesforce integrations can each have their own rate-limiting policies. Production tools need appropriate backoff alongside handling for timeouts, outages, input errors, and useful error messages; a concurrency guard alone does not make a tool reliable. section 3.3.4
