liveBook · Manning

Master weight

As explained in Build a DeepSeek Model (From Scratch)

A master weight is a high-precision persistent copy of a neural-network parameter used as the source for optimization updates.

A master weight is the persistent, high-precision parameter copy used for optimization updates. Mixed-precision training converts it to FP8 for fast matrix multiplication, while retaining FP32 or BF16 precision for the stored parameter and FP32 for critical accumulation. section 5.6.1

A linear layer uses the master weight through a precision-aware path:

  • Convert weights and BF16 inputs to FP8 on the fly for the forward multiplication. section 5.6.1
  • Accumulate multiplication results in FP32, then convert outputs back to BF16 for storage. section 5.6.1

Precision-aware linear layer

Keep W_master in FP32, cast it to FP8 for the matrix multiplication, accumulate the result in FP32, and update W_master with the computed weight gradient. section 5.6.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains precision assignments through forward computation, gradient calculation, accumulation, and the final update.
Examples
Uses FP8 matrix multiplication, BF16 activations and storage, FP32 accumulation, and an explicit master-weight update equation.
What it assumes
Assumes familiarity with neural-network layers, forward and backward propagation, gradients, and numerical formats.
What you can do after
Operate/configure a mixed-precision linear layer while keeping optimization updates in high-precision master parameters.
How it connects
Connects master weights to mixed-precision training, FP8 quantization, linear layers, gradients, and optimization updates.

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.