FP8 is an 8-bit floating-point format used to reduce the memory and compute costs of neural-network operations. Unlike INT8, it retains a sign, exponent, and mantissa, trading some precision for the flexibility to represent values across a range of magnitudes. Lower precision can cause numerical instability, so scaling and selective use of higher precision are important. section 5.5.3 section 1.2.1
The exponent controls dynamic range, while the mantissa controls precision. E5M2 allocates five bits to the exponent and two to the mantissa; E4M3 allocates four and three, respectively. E5M2 offers a wider range, whereas E4M3 provides greater precision. section 5.6.4
Mixed precision in a linear layer
For a forward pass, activations typically stored in BF16 and weights maintained in FP32 or BF16 are converted to FP8 for multiplication. Sensitive components, including master weights and optimizer states, remain in higher precision to support stable training. FP8 matrix multiplication requires hardware support, such as FP8-capable Tensor Cores. section 5.6.1 section 5.7
Figure 5.22 The mixed-precision framework with FP8 data format in the Linear operator. The diagram illustrates the flow of data through a linear layer, showing how inputs and weights are converted to low-precision FP8 for fast computation, while critical components like master weights and gradients are maintained in high-precision FP32 for stability.

Trace the precision changes through a linear layer, from FP8 computation to higher-precision training components. section 5.6.1
A common mixed-precision choice uses E4M3 for the forward pass and the wider-range E5M2 for backward gradients. DeepSeek instead uses E4M3 for both forward and backward FP8 operations. Fine-grained quantization makes this possible by scaling small blocks independently, limiting the effect of an outlier on values in other blocks. section 5.6.4
Other parts of the FP8 training framework address precision loss and changing data:
- Accumulate intermediate results in FP32 after low-precision Tensor Core computation to reduce underflow errors. section 5.7
- Calculate scaling factors online from the current batch rather than relying on historical data. section 5.7

