liveBook · Manning

FP32 register

As explained in Build a DeepSeek Model (From Scratch)

An FP32 register is a hardware register capable of storing a 32-bit floating-point value.

An FP32 register is a hardware register on a CUDA Core that stores a 32-bit floating-point value. In low-precision matrix multiplication, it receives partial sums periodically transferred from a Tensor Core's low-precision accumulator. Continuing accumulation in FP32 reduces underflow risk while preserving the speed of burst-oriented FP8 computation. section 5.6.3

The accumulation path separates throughput from numerical stability:

  • Tensor Cores execute WGMMA steps that multiply FP8 inputs and maintain a running low-precision partial sum. section 5.6.3
  • At a fixed interval Nc, a CUDA Core promotes the partial sum into an FP32 register. section 5.6.3
  • The FP32 register performs the final high-precision accumulation, limiting numerical errors from low-precision storage. section 5.6.3
Figure 5.31 The Increasing Accumulation Precision mechanism in detail. Low-precision accumulation happens in bursts on the Tensor Core, and results are periodically promoted to a high-precision FP32 register on the CUDA Core.

Trace how low-precision Tensor Core bursts periodically promote their results to an FP32 register on a CUDA Core. section 5.6.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Traces periodic promotion from a low-precision accumulator to FP32, but does not specify register layout or instruction-level implementation details.
Examples
Uses the y = Wx dot-product path and diagrams showing WGMMA bursts followed by periodic FP32 promotion.
What you can do after
Evaluate how periodic FP32 accumulation reduces underflow risk in FP8 matrix-multiplication pipelines.
How it connects
Connects Tensor Cores, CUDA Cores, WGMMA, FP8 quantization, matrix multiplication, and low-precision accumulation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.