Exponential backoff progressively delays retries after transient failures or service throttling, giving overloaded networks and APIs time to recover. section 2.2.6 section 3.3.4
Use backoff as part of a broader asynchronous reliability pattern:
- Set LiteLLM's num_retries parameter to retry failed requests automatically, such as num_retries=3. section 2.2.6
- Combine retries with asyncio.Semaphore so concurrency limits prevent large batches from overwhelming the API. section 2.2.6
- Collect results with return_exceptions=True so one request failure does not cancel successful concurrent calls. section 2.2.6
- Add backoff, timeout handling, input validation, and meaningful errors when turning a simple custom tool into a production-ready integration. section 3.3.4
Retrying a batch of LLM calls
Create 100 asynchronous tasks, allow only 10 API calls at once with asyncio.Semaphore(10), and pass num_retries=3 to acompletion so temporary failures receive automatic exponential backoff. section 2.2.6
Listing 2.9 Rate-limited concurrent requests
import asyncio
from litellm import acompletion
# Limit to 10 concurrent requests
semaphore = asyncio.Semaphore(10)
async def call_llm(prompt: str) -> str:
"""LLM call with rate limiting and automatic retry."""
async with semaphore:
response = await acompletion(
model="gpt-5-mini",
messages=[{"role": "user", "content": prompt}],
num_retries=3 # Automatic retry with exponential backoff
)
return response.choices[0].message.content
# Even with 100 concurrent tasks, only 10 API calls run at a time
prompts = [f"What is {i} + {i}?" for i in range(100)]
tasks = [call_llm(p) for p in prompts]
results = await asyncio.gather(*tasks, return_exceptions=True)
Inspect the complete Python pattern for combining a semaphore, LiteLLM retries, exponential backoff, and exception-tolerant result collection. section 2.2.6
