liveBook · Manning

Bfloat16

As explained in Quantization and Fast Inference

A 16-bit floating-point format with 8 exponent bits and 7 fraction bits, giving it a dynamic range similar to FP32.

bfloat16 (BF16) is a 16-bit floating-point format that trades precision for range. It uses 8 exponent bits and 7 fraction bits, preserving FP32’s dynamic range while storing each value in 2 bytes. Its maximum value is roughly 3.4 × 10³⁸. section 1.1 section 2.1

IEEE FP16 also uses 2 bytes, but allocates 5 bits to the exponent and 10 to the fraction. It has a finer mantissa grid—1,024 steps versus BF16’s 128—but tops out at about 65,504. BF16’s wider range helps accommodate gradients that would overflow FP16, at the cost of coarser rounding. section 2.1

BF16 has been a default format for large language model training since roughly 2020. A BF16-trained model has encountered coarse mantissa rounding throughout training; quantizing it to INT8 is often gentler than quantizing an FP16-trained model. INT8’s 256 levels and BF16’s floating-point mantissa steps are not the same kind of grid, so the comparison is a practical tendency rather than a guarantee. section 1.1 section 2.1

Checkpoint storage and traffic

Loading a Hugging Face checkpoint with torch_dtype=torch.bfloat16 gives BF16 weights. For a 7-billion-parameter model, those weights occupy roughly 14 GB, the same as FP16 weights. At batch size 1, reading the weights for each generated token therefore incurs the same weight-traffic volume in either 16-bit format. section 1.1

How Quantization and Fast Inference covers it

How deep it goes
Explains the exponent–mantissa tradeoff with range and precision comparisons, without implementing BF16 arithmetic.
Examples
Uses a 7-billion-parameter model and Hugging Face checkpoint loading to illustrate BF16 weight storage.
What you can do after
Evaluate the range, precision, and storage tradeoffs between BF16 and FP16 for model training and inference.
How it connects
Connects BF16 to FP32, FP16, gradient overflow, INT8 quantization, and memory traffic.

Also explained in

Compare how 6 books explain bfloat16 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.