Load testing checks whether a service handles its expected workload without unacceptable degradation or failure. Run it before production release or predictable traffic peaks, especially when requests are expensive and resource-intensive. section 6.3.2
A practical load test follows this sequence:
- Deploy the service in staging, send requests periodically, increase demand until failure or autoscaling, record metrics, and analyze the results. section 6.3.2
- Track latency and throughput at failure to estimate replica requirements; for streaming responses, also measure time to first token and tokens per second. section 6.3.2
Streaming service test
A Locust user posts a streaming prompt to /generate, counts returned lines, calculates latency, TTFT, and TPS, and writes the measurements to CSV. A run increasing to 50 users at one user per second peaks at 34 requests per second, exposing a service bottleneck.
