liveBook · Manning

Gpu core

As explained in Build a DeepSeek Model (From Scratch)

A CUDA core is a general-purpose arithmetic processing unit in an NVIDIA GPU that supports a broad range of computations.

A CUDA core is a general-purpose arithmetic unit in an NVIDIA GPU. Unlike a Tensor Core, it trades specialized matrix-multiplication speed for flexible computation and full FP32 precision. section 5.6.3

Promotion to CUDA Cores addresses precision loss during large low-precision matrix multiplications. Tensor Cores accumulate WGMMA results in bursts, then periodically transfer partial sums to CUDA cores for higher-precision FP32 accumulation. section 5.6.3

Accumulating a large matrix product

For an inner dimension k of 4096 or more, thousands of small products can accumulate numerical error. Periodic promotion limits reliance on the Tensor Core's low-precision accumulator; without sufficient precision, errors can approach 2%. section 5.6.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains CUDA Core and Tensor Core roles, WGMMA bursts, periodic promotion, and FP32 accumulation, but stops short of implementation code.
Examples
Uses k≥4096 matrix products, WGMMA accumulation bursts, periodic promotion, and potential errors approaching 2%.
What you can do after
Evaluate how periodic CUDA Core promotion can improve accumulation precision in low-precision matrix multiplication.
How it connects
Connects CUDA cores with Tensor Cores, matrix multiplication, WGMMA, FP32 precision, and numerical underflow.

Also explained in

Compare how 2 books explain gpu core →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.