E4M3 is an 8-bit floating-point encoding with one sign bit, four exponent bits, and three explicitly stored mantissa bits. Compared with E5M2, it gives up dynamic range to preserve higher precision. section 5.6.4 section 5.5.3
E4M3 supports an implementation strategy built around local scaling:
- Use fine-grained quantization to calculate independent scaling factors for small activation and weight blocks, limiting the effect of outliers. section 5.6.4
- Use E4M3 uniformly for forward and backward operations instead of pairing E4M3 forward computation with E5M2 gradients. section 5.6.4
- Retain high precision for sensitive components such as master weights and optimizer states while performing core computation in low precision. section 5.7
Uniform E4M3 training
A conventional mixed-precision design uses E4M3 for the forward pass and E5M2 for backward gradients, whose outliers may exceed E4M3’s range. With block-local scaling, uniform E4M3 becomes practical because each block manages its own dynamic range. section 5.6.4

