liveBook · Manning

Promotion to CUDA Cores

As explained in Build a DeepSeek Model (From Scratch)

Promotion to CUDA Cores is a mixed-precision accumulation technique that periodically transfers partial sums from low-precision Tensor Core computation to high-precision CUDA Core computation.

Promotion to CUDA Cores addresses numerical error in large low-precision matrix multiplications. When thousands of small products accumulate in a limited-precision register, underflow can distort the final result by as much as 2%. section 5.6.3

Tensor Cores perform fast, low-precision WGMMA multiplication and accumulation in bursts. At a fixed interval Nc, the partial sum is transferred to a CUDA Core and accumulated in a high-precision FP32 register. This periodic promotion preserves much of Tensor Core throughput while improving accumulation precision. section 5.6.3

Figure 5.30 The core strategy for increasing accumulation precision: intermediate results are periodically moved from the low-precision environment of the Tensor Cores to the high-precision environment of the CUDA Cores.

Visualizes the transfer path from low-precision Tensor Core accumulation to high-precision CUDA Core accumulation. section 5.6.3

Figure 5.31 The Increasing Accumulation Precision mechanism in detail. Low-precision accumulation happens in bursts on the Tensor Core, and results are periodically promoted to a high-precision FP32 register on the CUDA Core.

Details burst-based low-precision accumulation and periodic promotion into an FP32 CUDA Core register. section 5.6.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains WGMMA bursts, low-precision registers, periodic promotion at Nc, and final FP32 accumulation.
Examples
Uses the matrix-product accumulation formula and Figures 5.30–5.31 to depict periodic promotion.
What it assumes
Assumes matrix multiplication and GPU computational units while grounding Tensor Core and CUDA Core precision differences.
What you can do after
Evaluate how periodic FP32 promotion reduces accumulation error in low-precision matrix multiplication.
How it connects
Connects Tensor Cores, CUDA Cores, WGMMA, FP32, and mixed-precision matrix multiplication.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.