Energy per token measures the energy consumed to generate one output token. For GPU serving, divide average GPU power in watts by generation throughput in tokens per second; the result is joules per token. Higher throughput can reduce energy per token even when power draw stays constant. section 12.3.1
FP16 versus four-bit AWQ on an L4
At peak concurrency, FP16 uses 72.0 W and generates 1,587 tokens/s: 72.0 / 1,587 ≈ 45.4 mJ/token. AWQ uses 71.7 W and generates 2,926 tokens/s: 71.7 / 2,926 ≈ 24.5 mJ/token. AWQ consumes about 54% as much GPU energy per token, a 1.85× improvement driven by throughput rather than lower power. section 12.3.1
Table 12.1 Generation throughput, time-to-first-token p95, and inter-token latency p95 across the concurrency sweep on the same L4. ITL is inter-token latency
|
Concurrency
|
FP16 tokens/s
|
AWQ tokens/s
|
FP16 TTFT p95 (ms)
|
AWQ TTFT p95 (ms)
|
FP16 ITL p95 (ms)
|
AWQ ITL p95 (ms)
|
|
1
|
22.6
|
58.7
|
858.6
|
37.8
|
48.8
|
9.7
|
|
2
|
57.7
|
166.9
|
217.0
|
39.7
|
48.8
|
10.1
|
|
4
|
127.8
|
305.5
|
98.0
|
39.3
|
48.8
|
20.9
|
|
8
|
236.8
|
545.2
|
97.2
|
55.5
|
48.8
|
23.5
|
|
16
|
494.2
|
1,161.9
|
98.0
|
58.4
|
48.8
|
24.1
|
|
32
|
858.2
|
1,854.6
|
220.0
|
78.2
|
48.8
|
24.3
|
|
64
|
1,587.3
|
2,925.8
|
236.9
|
132.8
|
48.9
|
24.7
|
The concurrency sweep shows how throughput and latency change across FP16 and AWQ; the energy calculation above uses the peak-throughput row. section 12.3.1
Memory traffic is a major energy cost during token generation. An INT8 multiply-add costs roughly 0.2–0.5 pJ, while an HBM or off-chip DRAM access costs roughly 300–1,000 pJ. Reducing weight precision can cut the bytes read and free memory bandwidth. As batching increases, compute can eventually replace weight traffic as the limit. section 1.1 section 12.3.1
Check the measurement boundary and workload before comparing results. GPU power divided by generation throughput measures GPU energy per output token, not necessarily total system energy. It is not directly comparable to an estimated energy per image classification. Multiply energy per token by tokens served and the electricity rate to estimate the corresponding electricity cost. section 12.3.1 section 12.3.4
Treat lower energy as a hardware-dependent tradeoff, not a guaranteed quantization benefit. The AWQ result requires a supported vLLM four-bit kernel; below its hardware floor, the artifact takes a full-precision path. In the measured comparison, the AWQ canary achieved 89.9% greedy agreement with FP16, so validate quality alongside energy and latency. section 12.3.1 section 12.3.4
