An enterprise application calling a cloud-hosted LLM must control how quickly it sends requests. A rate limit caps request frequency, commonly as requests per second or minute. When a service enforces that limit by slowing or restricting requests, it is throttling. Both help prevent overload and preserve performance for other users. section 11.4.5 section 11.4.6
Do not confuse a rate limit with a quota. A quota caps resource consumption over a period; a rate limit controls how frequently requests arrive. For LLM deployments, token-based quotas may be allocated by model and region, while a requests-per-minute limit can still be hit if traffic arrives in a burst. Under pay-as-you-go pricing, token consumption also affects cost. section 11.4.5 section 11.4.4
Allocate capacity across deployments
With a 240,000-tokens-per-minute quota for one model in one region, an enterprise could assign 240K TPM to one deployment or 120K TPM to each of two deployments. That allocation does not remove the need to watch request frequency and spread requests over time. section 11.4.5 section 11.4.6
Manage limits as part of production operations:
- Check the limits for each model and region, then monitor consumption against assigned quotas so throttling does not come as a surprise. section 11.4.6
- Distribute requests evenly and ramp workloads gradually rather than creating sharp spikes. section 11.4.6
- Handle rate-limit errors with a pause and retry; adjust TPM allocation for busy deployments or request a quota increase if limits are consistently exceeded. section 11.4.6
- Use rate limiting alongside resource monitoring and input validation to reduce model denial-of-service risk; it is one control, not a substitute for security testing. section 13.2.3
from tenacity import (
retry,
stop_after_attempt,
wait_random_exponential,
)
@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))
def completion_backoff(conversation):
response = client.chat.completions.create(
model=MODEL,
messages=conversation,
temperature=TEMPERATURE,
max_tokens=MAX_TOKENS,
)
return response
The Python example wraps a chat-completion call with Tenacity retries. Its randomized exponential wait ranges from 1 to 60 seconds and stops after six attempts, illustrating bounded recovery rather than unlimited retries. section 11.4.6
Rate limits belong in the same operating plan as quotas, caching, load balancing, auto-scaling, and token-cost monitoring. Choose controls from observed traffic and service limits, then verify that bursts and retries preserve a usable experience. section 11.5.1 section 11.4.6
