liveBook · Manning

Weight gradient

As explained in Build a DeepSeek Model (From Scratch)

A weight gradient is the derivative of a training loss with respect to a neural-network weight parameter.

A weight gradient, dL/dW, measures how the training loss changes with a layer's weight matrix. It determines the direction and magnitude of each weight update. section 5.6.1

For a linear layer, the mixed-precision path separates fast computation from stable parameter updates:

  • Reuse the forward-pass input x, stored in FP8, and convert the incoming gradient dL/dz from BF16 to FP8. section 5.6.1
  • Compute dL/dW = xᵀ * (dL/dz), using an FP32 accumulator. section 5.6.1
  • Store dL/dW in FP32 because imprecise gradients can destabilize updates to the model's parameters. section 5.6.1
  • Let the optimizer update FP32 master weights while retaining optimizer states such as AdamW momentum and variance in FP32. section 5.6.1

One training iteration

After computing dL/dW in FP32, apply W_master_new = W_master_old - learning_rate * (dL/dW). Convert the updated master weights to FP8 for the next forward pass. section 5.6.1

Figure 5.22 The mixed-precision framework with FP8 data format in the Linear operator. The diagram illustrates the flow of data through a linear layer, showing how inputs and weights are converted to low-precision FP8 for fast computation, while critical components like master weights and gradients are maintained in high-precision FP32 for stability.

Trace how FP8 inputs and weights support fast GEMMs while FP32 gradients, master weights, and optimizer states preserve update stability. section 5.6.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains the Wgrad equation, operand conversions, FP32 accumulation, storage, and optimizer update, without deriving backpropagation.
Examples
Uses dL/dW = xᵀ * (dL/dz), FP8 operand conversion, and the FP32 master-weight update equation.
What you can do after
Apply mixed-precision training by computing and storing weight gradients in FP32 before updating master weights.
How it connects
Connects weight gradients to backpropagation, mixed-precision training, FP8 quantization, master weights, and optimizer states.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.