liveBook · Manning

E4M3

As explained in Build a DeepSeek Model (From Scratch)

E4M3 is an FP8 encoding with four exponent bits and three explicitly stored mantissa bits.

E4M3 is an 8-bit floating-point encoding with one sign bit, four exponent bits, and three explicitly stored mantissa bits. Compared with E5M2, it gives up dynamic range to preserve higher precision. section 5.6.4 section 5.5.3

E4M3 supports an implementation strategy built around local scaling:

  • Use fine-grained quantization to calculate independent scaling factors for small activation and weight blocks, limiting the effect of outliers. section 5.6.4
  • Use E4M3 uniformly for forward and backward operations instead of pairing E4M3 forward computation with E5M2 gradients. section 5.6.4
  • Retain high precision for sensitive components such as master weights and optimizer states while performing core computation in low precision. section 5.7

Uniform E4M3 training

A conventional mixed-precision design uses E4M3 for the forward pass and E5M2 for backward gradients, whose outliers may exceed E4M3’s range. With block-local scaling, uniform E4M3 becomes practical because each block manages its own dynamic range. section 5.6.4

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains bit allocation, precision–range trade-offs, and training-format selection, without detailing encoding equations.
Examples
Contrasts E4M3 with E5M2 and INT8, then applies it to uniformly quantized forward and backward training operations.
What it assumes
Uses the distinction between floating-point exponent range and mantissa precision as its conceptual foundation.
What you can do after
Evaluate FP8 format choices and apply uniform E4M3 when block-local scaling sufficiently limits outlier-driven range loss.
How it connects
Connects E4M3 to FP8, E5M2, fine-grained quantization, mixed-precision training, and forward and backward passes.

Also explained in

Compare how 2 books explain E4M3 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.