bfloat16 (BF16) is a 16-bit floating-point format that trades precision for range. It uses 8 exponent bits and 7 fraction bits, preserving FP32’s dynamic range while storing each value in 2 bytes. Its maximum value is roughly 3.4 × 10³⁸. section 1.1 section 2.1
IEEE FP16 also uses 2 bytes, but allocates 5 bits to the exponent and 10 to the fraction. It has a finer mantissa grid—1,024 steps versus BF16’s 128—but tops out at about 65,504. BF16’s wider range helps accommodate gradients that would overflow FP16, at the cost of coarser rounding. section 2.1
BF16 has been a default format for large language model training since roughly 2020. A BF16-trained model has encountered coarse mantissa rounding throughout training; quantizing it to INT8 is often gentler than quantizing an FP16-trained model. INT8’s 256 levels and BF16’s floating-point mantissa steps are not the same kind of grid, so the comparison is a practical tendency rather than a guarantee. section 1.1 section 2.1
Checkpoint storage and traffic
Loading a Hugging Face checkpoint with torch_dtype=torch.bfloat16 gives BF16 weights. For a 7-billion-parameter model, those weights occupy roughly 14 GB, the same as FP16 weights. At batch size 1, reading the weights for each generated token therefore incurs the same weight-traffic volume in either 16-bit format. section 1.1





