liveBook · Manning

FP32

As explained in Build a DeepSeek Model (From Scratch)

Float32, also called FP32, is a 32-bit floating-point numerical format that uses four bytes per value.

FP32, or float32, is a 32-bit floating-point format that stores each value in four bytes. Its high precision and wide dynamic range make it a useful baseline for model parameters and numerically sensitive computations, but its memory cost motivates lower-precision alternatives. section 5.5.1 section 2.5.1

An FP32 value allocates its bits as follows:

  • 1 sign bit distinguishes positive from negative values. section 5.5.3
  • 8 exponent bits control magnitude and dynamic range. section 5.5.3
  • 23 mantissa bits provide precision: more mantissa bits allow more significant digits and smaller gaps between representable numbers. section 5.5.3
Figure 5.15 A visual comparison of a number represented in 32-bit floating-point (FP32) versus 16-bit floating-point (FP16). The diagram illustrates the reduction in the number of bits allocated for the exponent and the mantissa, which results in lower precision and a smaller memory footprint.

Compare the bit allocation and memory footprint of FP32 with FP16. section 5.5.1

Reducing bit width saves memory but changes the numerical trade-off. FP16 uses two bytes per value and has fewer exponent and mantissa bits than FP32, reducing both range and precision. BF16 also uses two bytes, but retains FP32's eight exponent bits while reducing the mantissa to seven bits. section 5.5.3 section 2.5.1

A mixed-precision linear layer

Inputs and weights can be converted to FP8 for matrix multiplication, while the result is accumulated in FP32 and then converted to BF16 for storage. A master weight copy can be maintained in FP32 for sensitive updates. This reserves higher precision for stability while reducing the cost of the heavy multiplication. section 5.6.1

Accumulation itself can be a bottleneck: summing thousands of FP8 products in a limited-precision Tensor Core accumulator can introduce substantial error. One strategy periodically promotes intermediate results to FP32 on CUDA Cores for higher-precision final accumulation. section 5.6.3

Figure 5.31 The Increasing Accumulation Precision mechanism in detail. Low-precision accumulation happens in bursts on the Tensor Core, and results are periodically promoted to a high-precision FP32 register on the CUDA Core.

Follow the transfer from low-precision Tensor Core accumulation to an FP32 register on a CUDA Core. section 5.6.3

PyTorch automatic mixed precision

Configure torch.amp.autocast to use bfloat16 or float16 on CUDA. The training setup retains FP32 master weights and uses FP32 for sensitive operations such as reductions, softmax, and loss, while lower precision handles much of the compute. section 6.3.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains sign, exponent, and mantissa allocation, then traces FP32 through mixed-precision computation and accumulation.
Examples
Uses an FP8 linear layer, CUDA Core accumulation promotion, and a PyTorch autocast training configuration.
What you can do after
Apply mixed precision via PyTorch autocast while retaining FP32 for master weights and sensitive operations.
How it connects
Connects FP32 to quantization, mixed precision, floating-point range, numerical precision, and accumulation.

Also explained in

Compare how 6 books explain FP32 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.