liveBook · Manning

FP8

As explained in Build a DeepSeek Model (From Scratch)

An 8-bit floating-point numerical format used to represent and compute neural-network values with reduced precision.

FP8 is an 8-bit floating-point format used to reduce the memory and compute costs of neural-network operations. Unlike INT8, it retains a sign, exponent, and mantissa, trading some precision for the flexibility to represent values across a range of magnitudes. Lower precision can cause numerical instability, so scaling and selective use of higher precision are important. section 5.5.3 section 1.2.1

The exponent controls dynamic range, while the mantissa controls precision. E5M2 allocates five bits to the exponent and two to the mantissa; E4M3 allocates four and three, respectively. E5M2 offers a wider range, whereas E4M3 provides greater precision. section 5.6.4

Mixed precision in a linear layer

For a forward pass, activations typically stored in BF16 and weights maintained in FP32 or BF16 are converted to FP8 for multiplication. Sensitive components, including master weights and optimizer states, remain in higher precision to support stable training. FP8 matrix multiplication requires hardware support, such as FP8-capable Tensor Cores. section 5.6.1 section 5.7

Figure 5.22 The mixed-precision framework with FP8 data format in the Linear operator. The diagram illustrates the flow of data through a linear layer, showing how inputs and weights are converted to low-precision FP8 for fast computation, while critical components like master weights and gradients are maintained in high-precision FP32 for stability.

Trace the precision changes through a linear layer, from FP8 computation to higher-precision training components. section 5.6.1

A common mixed-precision choice uses E4M3 for the forward pass and the wider-range E5M2 for backward gradients. DeepSeek instead uses E4M3 for both forward and backward FP8 operations. Fine-grained quantization makes this possible by scaling small blocks independently, limiting the effect of an outlier on values in other blocks. section 5.6.4

Other parts of the FP8 training framework address precision loss and changing data:

  • Accumulate intermediate results in FP32 after low-precision Tensor Core computation to reduce underflow errors. section 5.7
  • Calculate scaling factors online from the current batch rather than relying on historical data. section 5.7

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains format trade-offs and the mixed-precision training framework, including scaling, accumulation, and forward and backward format choices.
Examples
Traces a linear layer’s mixed-precision data flow and compares E4M3 with E5M2.
What you can do after
Evaluate and understand FP8 precision choices and the scaling strategies used to support stable training.
How it connects
Connects FP8 to mixed-precision training, fine-grained quantization, dynamic range, and numerical stability.

Also explained in

Compare how 2 books explain FP8 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.