liveBook · Manning

Accumulator

As explained in Build a DeepSeek Model (From Scratch)

An accumulator is a hardware register that stores a running sum of products during a computation.

An accumulator is a hardware register that stores the running sum of products in a matrix-multiplication dot product. For Y = WX, each product WᵢₗXₗⱼ is added into the accumulator until one output element is complete. With FP8 inputs, limited accumulator precision becomes a hardware bottleneck: large inner dimensions such as k = 4096 require thousands of additions, and insufficient precision can cause underflow and substantial numerical error. section 5.6.3

Y_ij = W_i1*X_1j + W_i2*X_2j + ... + W_ik*X_kj

The expanded dot-product equation makes each accumulation step explicit. section 5.6.3

Increasing accumulation precision separates fast low-precision multiplication from more precise summation. Tensor Cores accumulate products in bursts, then periodically promote intermediate results to an FP32 register on CUDA Cores. This uses specialized units for throughput while reducing error from prolonged low-precision accumulation. section 5.6.3

Figure 5.30 The core strategy for increasing accumulation precision: intermediate results are periodically moved from the low-precision environment of the Tensor Cores to the high-precision environment of the CUDA Cores.

The figure traces intermediate results moving from the low-precision Tensor Core environment to higher-precision CUDA Core storage. section 5.6.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Defines the register, derives its dot-product role, and explains periodic precision promotion without detailing implementation code.
Examples
Uses Y = WX, thousands of products at k = 4096, and Tensor Core bursts promoted into FP32 registers.
What it assumes
Assumes familiarity with matrix multiplication, dot products, FP8 values, and GPU compute units.
What you can do after
Diagnose numerical error from insufficient accumulation precision and understand promotion between GPU compute units.
How it connects
Connects accumulators with matrix multiplication, dot products, FP8, Tensor Cores, CUDA Cores, and FP32.

Also explained in

Compare how 7 books explain accumulator →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.