A master weight is the persistent, high-precision parameter copy used for optimization updates. Mixed-precision training converts it to FP8 for fast matrix multiplication, while retaining FP32 or BF16 precision for the stored parameter and FP32 for critical accumulation. section 5.6.1
A linear layer uses the master weight through a precision-aware path:
- Convert weights and BF16 inputs to FP8 on the fly for the forward multiplication. section 5.6.1
- Accumulate multiplication results in FP32, then convert outputs back to BF16 for storage. section 5.6.1
Precision-aware linear layer
Keep W_master in FP32, cast it to FP8 for the matrix multiplication, accumulate the result in FP32, and update W_master with the computed weight gradient. section 5.6.1
