FP32, or float32, is a 32-bit floating-point format that stores each value in four bytes. Its high precision and wide dynamic range make it a useful baseline for model parameters and numerically sensitive computations, but its memory cost motivates lower-precision alternatives. section 5.5.1 section 2.5.1
An FP32 value allocates its bits as follows:
- 1 sign bit distinguishes positive from negative values. section 5.5.3
- 8 exponent bits control magnitude and dynamic range. section 5.5.3
- 23 mantissa bits provide precision: more mantissa bits allow more significant digits and smaller gaps between representable numbers. section 5.5.3
Figure 5.15 A visual comparison of a number represented in 32-bit floating-point (FP32) versus 16-bit floating-point (FP16). The diagram illustrates the reduction in the number of bits allocated for the exponent and the mantissa, which results in lower precision and a smaller memory footprint.

Compare the bit allocation and memory footprint of FP32 with FP16. section 5.5.1
Reducing bit width saves memory but changes the numerical trade-off. FP16 uses two bytes per value and has fewer exponent and mantissa bits than FP32, reducing both range and precision. BF16 also uses two bytes, but retains FP32's eight exponent bits while reducing the mantissa to seven bits. section 5.5.3 section 2.5.1
A mixed-precision linear layer
Inputs and weights can be converted to FP8 for matrix multiplication, while the result is accumulated in FP32 and then converted to BF16 for storage. A master weight copy can be maintained in FP32 for sensitive updates. This reserves higher precision for stability while reducing the cost of the heavy multiplication. section 5.6.1
Accumulation itself can be a bottleneck: summing thousands of FP8 products in a limited-precision Tensor Core accumulator can introduce substantial error. One strategy periodically promotes intermediate results to FP32 on CUDA Cores for higher-precision final accumulation. section 5.6.3
Figure 5.31 The Increasing Accumulation Precision mechanism in detail. Low-precision accumulation happens in bursts on the Tensor Core, and results are periodically promoted to a high-precision FP32 register on the CUDA Core.

Follow the transfer from low-precision Tensor Core accumulation to an FP32 register on a CUDA Core. section 5.6.3
PyTorch automatic mixed precision
Configure torch.amp.autocast to use bfloat16 or float16 on CUDA. The training setup retains FP32 master weights and uses FP32 for sensitive operations such as reductions, softmax, and loss, while lower precision handles much of the compute. section 6.3.1





