Tokens per minute (TPM) measures how many tokens a language-model service processes per minute. Under PAYGO, token consumption follows prompt and response length, so costs rise and fall with usage. section 11.4.4
Cloud providers use TPM as a quota and capacity measure for model deployments. Allocating TPM to one deployment reduces the quota available to others, making distribution important for variable or high-traffic workloads. section 11.4.4 section 11.4.5
Production teams manage TPM by:
- Increasing TPM for high-traffic deployments and reducing it for deployments with limited demand. section 11.4.6
- Monitoring quotas and requesting increases when sustained demand exceeds available capacity. section 11.4.6 section 11.4.5
- Combining PAYGO for variable workloads with provisioned throughput units (PTUs) for steady performance requirements. section 11.4.6 section 11.4.3
Quota allocation
With a 240,000 TPM quota for one model in a region, an enterprise could assign 240K TPM to one deployment, 120K TPM to each of two deployments, or distribute a smaller total across several deployments. section 11.4.5
from tenacity import (
retry,
stop_after_attempt,
wait_random_exponential,
)
@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))
def completion_backoff(conversation):
response = client.chat.completions.create(
model=MODEL,
messages=conversation,
temperature=TEMPERATURE,
max_tokens=MAX_TOKENS,
)
return response
Use exponential backoff and bounded retries when quota or rate-limit pressure causes throttling. section 11.4.6
