liveBook · Manning

Tensor Core

As explained in Build a DeepSeek Model (From Scratch)

A Tensor Core is a GPU computational unit specialized for high-throughput matrix multiplication and accumulation, typically at reduced numerical precision.

Tensor Cores are specialized GPU units for high-throughput low-precision matrix multiplication and accumulation. With FP8 inputs, they compute products rapidly but may accumulate their running sum with limited precision. Large inner dimensions can therefore introduce underflow and significant numerical error. section 5.6.3

Increasing accumulation precision addresses the architectural trade-off as follows:

  • Compute FP8 products in bursts on Tensor Cores. section 5.6.3
  • Use promotion to reduce accumulation errors without abandoning Tensor Core throughput for the multiplication itself. section 5.6.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains products, accumulators, underflow, and periodic precision promotion, but stops short of implementation code.
Examples
Uses a dot-product formula, H800 hardware, inner dimension 4096, and figures showing burst accumulation with FP32 promotion.
What it assumes
Assumes matrix multiplication and dot products, while grounding the accumulator as the running sum of individual products.
What you can do after
Evaluate accumulation-precision trade-offs and understand why periodic promotion can protect low-precision matrix results.
How it connects
Connects matrix multiplication, FP8 quantization, accumulators, CUDA Cores, FP32 precision, and low-precision training.

Also explained in

Compare how 4 books explain Tensor Core →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.