float16, or FP16, trades dynamic range for precision in a 16-bit floating-point format. Its smaller representation can reduce model memory traffic, while FP16 arithmetic can use less energy than FP32 arithmetic. The tradeoff matters in inference, where moving data can cost much more energy than computing on it. section 1.1 section 2.1
IEEE FP16 allocates 5 bits to the exponent and 10 to the fraction, leaving one sign bit. Its fraction has 1,024 possible steps, and its largest finite value is about 65,504. Floating-point spacing varies with the exponent, giving smaller numbers a finer absolute grid. Values above FP16’s range can overflow to infinity. section 2.1
Table 2.2 Quantize only the mantissa to the nearest multiple of 1/16
|
True Value
|
(Approx) mantissa × 2^exp
|
Quantized mantissa
|
Reconstructed value
|
|
0.01
|
1.28 × 2⁻⁷
|
1.3125 × 2⁻⁷
|
0.01025390625
|
|
0.10
|
1.60 × 2⁻⁴
|
1.6250 × 2⁻⁴
|
0.1015625
|
|
1.00
|
1.00 × 2⁰
|
1.0000 × 2⁰
|
1.0
|
|
3.00
|
1.50 × 2¹
|
1.5000 × 2¹
|
3.0
|
A four-fraction-bit toy float illustrates exponent-dependent spacing; its values and precision are not FP16. section 2.1
BF16 makes the opposite tradeoff: 8 exponent bits and 7 fraction bits. Its 128-step fraction grid is eight times coarser, but its range extends to about 3.4 × 10³⁸. FP16 offers finer precision, while BF16 can accommodate much larger gradients in training. section 2.1
Energy and serving comparisons
Approximate FP16 fused multiply-add energy is 1–2 pJ, versus 3–5 pJ for FP32; an HBM or off-chip DRAM access is roughly 300–1,000 pJ. Exact costs vary by hardware. For a serving decision, compare FP16 against a lower-bit format on the same model and hardware: one capture uses Qwen2.5-3B-Instruct on an L4, comparing FP16 with AWQ four-bit across the same concurrency sweep. It does not establish results for GPTQ or FP8. section 1.1 section 12.3



