Concurrency measures how many requests or users a model instance serves simultaneously. It affects batching, per-user speed, and the point where memory limits cause requests to queue. section 10.1 section 12.3.2
Read concurrency alongside aggregate throughput and per-user throughput:
- On an L4, batching at concurrency 8 raises aggregate throughput above 200 tokens per second. CPU serving batches less effectively: above two concurrent users, per-user throughput falls faster than aggregate throughput rises. section 10.1
- Concurrency also tests memory headroom. A larger KV cache can hold more in-flight sequences before the scheduler queues requests, but a small model may never use enough memory for this advantage to appear. section 12.3.2
A sweep that never reaches saturation
For a 3B model on an L4, an FP16-versus-AWQ concurrency sweep kept queue depth at zero and KV-cache occupancy below about 1.3%. The freed-memory advantage did not change observed saturation because neither configuration approached its memory limit. section 12.3.2
Figure 10.1 Effective cost per million Llama-2-7B Q4_K_M tokens versus sustained demand, on Google Cloud c3-standard-22 (CPU-only, solid) and g2-standard-4 with one NVIDIA L4 (dashed) in us-central1 at on-demand list pricing. Each curve declines hyperbolically until the instance's single-stream throughput ceiling (CPU at 20 tok/s, GPU at 50 tok/s; tick marks at successive replica boundaries), then steps up as additional replicas absorb demand above ceiling. Below saturation the ratio between curves equals the ratio of hourly prices (1.57×). At the GPU's saturation the ratio is roughly 3.9×, widening further at higher demand because the CPU instance saturates at 2.5× lower throughput. The L4 wins on cost per token at every demand point in the plot; the gap is bounded between 1.57× and roughly 5×.

Compare how sustained demand and instance throughput ceilings affect serving cost as replicas are added. section 10.1

