bfloat16 (BF16) is a 16-bit floating-point format with 1 sign bit, 8 exponent bits, and 7 mantissa bits. Matching FP32's 8-bit exponent gives BF16 a wide dynamic range, while the shorter mantissa reduces memory use and precision. section 5.5.3
Use BF16 when range matters more than fine-grained numerical precision:
- Compared with FP16, BF16 is more resistant to overflow because it preserves FP32's exponent width, although its precision is lower. section 5.5.3
- Compared with FP32, BF16 halves storage from 32 to 16 bits while retaining the same exponent width. section 5.5.3
- Compared with FP8 or INT8, BF16 uses more memory but provides substantially more numerical precision or dynamic range. section 5.5.3
Mixed-precision training commonly stores activations in BF16 while converting inputs and weights to FP8 for matrix multiplication. Products accumulate in FP32, results return to BF16, and master weights and updates remain in FP32 for stability. section 5.6.1
KV-cache memory
A BF16 key-value cache uses 2 bytes per parameter. For layers l, batch size b, heads n, head size h, and sequence length s, its size is l × b × n × h × s × 2 × 2: one factor of 2 stores keys and values, and the other represents BF16's two bytes. section 2.5.1
Figure 5.19 A comparison of FP32 and BFloat16.

Visualizes BF16's shared exponent width with FP32 and reduced mantissa width. section 5.5.3





