liveBook · Manning

Float16

As explained in Quantization and Fast Inference

A 16-bit floating-point format with 5 exponent bits and 10 fraction bits.

float16, or FP16, trades dynamic range for precision in a 16-bit floating-point format. Its smaller representation can reduce model memory traffic, while FP16 arithmetic can use less energy than FP32 arithmetic. The tradeoff matters in inference, where moving data can cost much more energy than computing on it. section 1.1 section 2.1

IEEE FP16 allocates 5 bits to the exponent and 10 to the fraction, leaving one sign bit. Its fraction has 1,024 possible steps, and its largest finite value is about 65,504. Floating-point spacing varies with the exponent, giving smaller numbers a finer absolute grid. Values above FP16’s range can overflow to infinity. section 2.1

Table 2.2 Quantize only the mantissa to the nearest multiple of 1/16
True Value
(Approx) mantissa × 2^exp
Quantized mantissa
Reconstructed value
0.01
1.28 × 2⁻⁷
1.3125 × 2⁻⁷
0.01025390625
0.10
1.60 × 2⁻⁴
1.6250 × 2⁻⁴
0.1015625
1.00
1.00 × 2⁰
1.0000 × 2⁰
1.0
3.00
1.50 × 2¹
1.5000 × 2¹
3.0

A four-fraction-bit toy float illustrates exponent-dependent spacing; its values and precision are not FP16. section 2.1

BF16 makes the opposite tradeoff: 8 exponent bits and 7 fraction bits. Its 128-step fraction grid is eight times coarser, but its range extends to about 3.4 × 10³⁸. FP16 offers finer precision, while BF16 can accommodate much larger gradients in training. section 2.1

Energy and serving comparisons

Approximate FP16 fused multiply-add energy is 1–2 pJ, versus 3–5 pJ for FP32; an HBM or off-chip DRAM access is roughly 300–1,000 pJ. Exact costs vary by hardware. For a serving decision, compare FP16 against a lower-bit format on the same model and hardware: one capture uses Qwen2.5-3B-Instruct on an L4, comparing FP16 with AWQ four-bit across the same concurrency sweep. It does not establish results for GPTQ or FP8. section 1.1 section 12.3

How Quantization and Fast Inference covers it

How deep it goes
Explains exponent and fraction tradeoffs, numerical range, and relative energy costs without detailing FP16 encoding or implementation.
Examples
Illustrates floating-point rounding with a four-fraction-bit toy format and compares FP16 with AWQ four-bit serving on an L4.
What you can do after
Enables evaluation of FP16 precision, range, and energy tradeoffs when choosing training or inference formats.
How it connects
Connects FP16 to BF16, FP32, quantization, memory bandwidth, and mixed-precision training.

Also explained in

Compare how 4 books explain float16 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.