Throughput is the amount of work an LLM service successfully handles over a period of time. For a production application, it answers a practical question: can the deployment keep up with demand? Requests per second (RPS) describes incoming load, while throughput describes what the server handles. High RPS paired with low throughput can signal that the service is struggling. section 11.4.1
Read throughput alongside time to first token, time per output token, and end-to-end latency. An LLM processes prompt tokens together, then generates output tokens sequentially; request volume alone therefore does not describe the whole workload. Monitor response time, errors, and resource utilization too, so a throughput change can inform scaling or troubleshooting rather than stand as an isolated number. section 11.4.1 section 11.4.7
Sizing an Azure OpenAI deployment
Suppose an enterprise application needs more model-processing capacity. Azure OpenAI provisioned throughput units (PTUs) reserve capacity for prompts and completions; throughput per PTU varies by model and version. Estimate the workload shape with the Azure OpenAI Capacity calculator before choosing the PTU count for a provisioned-managed deployment. Compare that option with pay-as-you-go token consumption, then test under load: reserved capacity is a sizing choice, not a substitute for measuring the application's observed performance. section 11.4.3 section 11.5.1
For ongoing operation:
- Simulate multiple users with a tool such as Apache JMeter or Locust, and measure throughput and latency under stress. section 11.5.1
- Collect metrics continuously with tools such as Prometheus and Grafana; investigate declines alongside errors and resource use. section 11.5.1 section 11.4.7
- Consider caching, load balancing, and auto-scaling when demand rises; use queuing, throttling, and quotas to manage access and cost. section 11.5.1 section 11.4.3
