Concurrent LLM requests can reduce waiting time, but sending too many at once can trigger provider rate limits. A semaphore puts a ceiling on requests in progress without requiring tasks to run sequentially: tasks beyond the ceiling wait until a slot is available. section 2.2.6
Create an `asyncio.Semaphore(10)` and place `await acompletion(...)` inside `async with semaphore`. The block acquires a slot before the API call and releases it afterward. Even if `asyncio.gather` runs 100 tasks, no more than 10 enter the protected call at once. section 2.2.6
Listing 2.9 Rate-limited concurrent requests
import asyncio
from litellm import acompletion
# Limit to 10 concurrent requests
semaphore = asyncio.Semaphore(10)
async def call_llm(prompt: str) -> str:
"""LLM call with rate limiting and automatic retry."""
async with semaphore:
response = await acompletion(
model="gpt-5-mini",
messages=[{"role": "user", "content": prompt}],
num_retries=3 # Automatic retry with exponential backoff
)
return response.choices[0].message.content
# Even with 100 concurrent tasks, only 10 API calls run at a time
prompts = [f"What is {i} + {i}?" for i in range(100)]
tasks = [call_llm(p) for p in prompts]
results = await asyncio.gather(*tasks, return_exceptions=True)
The request wrapper keeps the concurrency limit around the call itself, while LiteLLM retries transient failures and `gather` returns individual exceptions for inspection. section 2.2.6
For evaluations spanning providers, use a separate semaphore per provider. A single shared limit could make a burst of requests to one provider hold up work destined for another. Provider limits of 30 for OpenAI and 10 for Anthropic are starting values, not universal settings; lower them when rate-limit errors occur or raise them when the applicable tier permits more concurrency. section 2.4.2
Listing 2.12 Provider-specific rate limiting
PROVIDER_SEMAPHORES = {
"openai": asyncio.Semaphore(30),
"anthropic": asyncio.Semaphore(10),
}
def get_provider(model: str) -> str:
"""Extract provider name from model string."""
return "anthropic" if model.startswith("anthropic/") else "openai"
The provider-to-semaphore mapping selects a concurrency limit from the model name. section 2.4.2
Keep retry and result handling alongside this limit. LiteLLM’s `num_retries` handles transient failures with backoff; `asyncio.gather(..., return_exceptions=True)` lets successful results survive individual failures. A semaphore controls simultaneous calls, while those other mechanisms address failures. section 2.2.6
