A weight gradient, dL/dW, measures how the training loss changes with a layer's weight matrix. It determines the direction and magnitude of each weight update. section 5.6.1
For a linear layer, the mixed-precision path separates fast computation from stable parameter updates:
- Reuse the forward-pass input x, stored in FP8, and convert the incoming gradient dL/dz from BF16 to FP8. section 5.6.1
- Compute dL/dW = xᵀ * (dL/dz), using an FP32 accumulator. section 5.6.1
- Store dL/dW in FP32 because imprecise gradients can destabilize updates to the model's parameters. section 5.6.1
- Let the optimizer update FP32 master weights while retaining optimizer states such as AdamW momentum and variance in FP32. section 5.6.1
One training iteration
After computing dL/dW in FP32, apply W_master_new = W_master_old - learning_rate * (dL/dW). Convert the updated master weights to FP8 for the next forward pass. section 5.6.1
Figure 5.22 The mixed-precision framework with FP8 data format in the Linear operator. The diagram illustrates the flow of data through a linear layer, showing how inputs and weights are converted to low-precision FP8 for fast computation, while critical components like master weights and gradients are maintained in high-precision FP32 for stability.

Trace how FP8 inputs and weights support fast GEMMs while FP32 gradients, master weights, and optimizer states preserve update stability. section 5.6.1
