Throughput measures completed work per unit of time. For language-model generation, decode throughput is commonly reported in tokens per second (tok/s); for other inference workloads, it may be inferences per second. Measure it under the workload and duration the deployment must sustain, not just in a short burst. section 10.1 section 11.3.4
Throughput is a capacity and cost metric, not a substitute for user-facing latency. Track time to first token and inter-token latency alongside generation rate; use end-to-end latency for a user-facing target. Compare rates only when the work units and measurement conditions are comparable. section 12.0 section 12.3.1 section 12.3.4
For bandwidth-bound decode, reading weights can dominate the time per token. Reducing weight size can increase throughput, but the result depends on the format, kernel, hardware, and batch size. On Apple M3 with Metal, one 64-token completion ran at 6.5 tok/s in F16 and 23.4 tok/s in IQ4_XS, a 3.6× increase; the ranking between IQ4_XS and Q4_K_M can shrink or reverse on older x86 CPUs. section 10.3.4 section 12.3.1
Concurrency changes the comparison
On the same NVIDIA L4, vLLM served the same model under matched load in FP16 and AWQ. At concurrency 1, generation throughput was 22.6 versus 58.7 tok/s; at concurrency 64, it was 1,587.3 versus 2,925.8 tok/s. Batching raised throughput for both, while AWQ inter-token latency p95 rose from 9.7 to 24.7 ms. Higher aggregate throughput does not by itself describe the latency of each request. section 12.3.1
Table 12.1 Generation throughput, time-to-first-token p95, and inter-token latency p95 across the concurrency sweep on the same L4. ITL is inter-token latency
|
Concurrency
|
FP16 tokens/s
|
AWQ tokens/s
|
FP16 TTFT p95 (ms)
|
AWQ TTFT p95 (ms)
|
FP16 ITL p95 (ms)
|
AWQ ITL p95 (ms)
|
|
1
|
22.6
|
58.7
|
858.6
|
37.8
|
48.8
|
9.7
|
|
2
|
57.7
|
166.9
|
217.0
|
39.7
|
48.8
|
10.1
|
|
4
|
127.8
|
305.5
|
98.0
|
39.3
|
48.8
|
20.9
|
|
8
|
236.8
|
545.2
|
97.2
|
55.5
|
48.8
|
23.5
|
|
16
|
494.2
|
1,161.9
|
98.0
|
58.4
|
48.8
|
24.1
|
|
32
|
858.2
|
1,854.6
|
220.0
|
78.2
|
48.8
|
24.3
|
|
64
|
1,587.3
|
2,925.8
|
236.9
|
132.8
|
48.9
|
24.7
|
Inspect the matched concurrency sweep to compare generation throughput with time-to-first-token and inter-token latency. section 12.3.1
Sustained performance can differ from the first measurement window. On a fanless M3 MacBook Air, an EfficientNet-Lite0 workload fell from roughly 2,660 to 2,360 inferences/s over five minutes, an 11% soft throttle. A different workload on the same device gained throughput as model-load and JIT costs faded. Measure the final interval of a thermally representative run rather than assuming that a burst rate will hold. section 11.3.4
At full utilization, cost per million generated tokens is hourly instance price × 1,000,000 ÷ (tok/s × 3,600). In one single-stream Llama-2-7B Q4_K_M comparison, a CPU VM at approximately 20 tok/s cost $15.40 per million tokens, versus $3.93 for an L4 instance estimated at 50 tok/s. Actual demand and replica boundaries affect effective cost; the L4 rate was an estimate, not a direct measurement. section 10.1
Validate throughput on the target host with the intended build, quantization, thread count, and workload. CPU build flags and thread choices can silently leave performance unused; on cloud VMs, a physical-core thread-count rule is not universally optimal. Also check memory and output quality: a faster format is not automatically an acceptable deployment. section 10.0 section 10.5 section 12.3.4































