E4M3 is an 8-bit floating-point format with one sign bit, four exponent bits, and three mantissa bits. It represents 253 finite values spanning ±448. Its narrow range makes scaling essential, while its extra mantissa bit relative to E5M2 gives better precision for weights and activations concentrated near zero. section 8.2 section 8.2.2
Figure 8.3 Near-zero density in [0, 0.5]. E4M3 packs 49 representable values into this interval; E5M2 packs 57 (more exponent range means more subnormal granularity); INT8 symmetric fits only 32. Neural network weight and activation distributions peak sharply near zero, so denser coverage here translates directly to lower quantization error where it matters most.

The near-zero comparison shows why E4M3's non-uniform spacing can reduce quantization error for neural-network values concentrated around zero. section 8.2.1
Configure scaling to keep values in range and limit outlier damage:
- Scale by the tensor's maximum absolute value before quantization, then multiply by the scale after dequantization; otherwise values beyond ±448 clip. section 8.2.2
- Use block scaling for activations when outliers distort a per-tensor scale. Blocks of 32 or 128 elements localize the effect; block-32 adds 12.5% storage overhead. section 8.2.2
- Use per-tensor scales for weights as a practical default; production inference runtimes may choose per-tensor or per-token activation scales. section 8.2.2
Weight-only inference
In an OPT-6.7B test, E4M3 weights stored as one byte each reduced model size from 12.40 GB to 6.21 GB. WikiText-2 perplexity remained 16.26, matching the BF16 result. Direct FP8 matmul requires FP8 tensor cores: the cited PyTorch kernel requires row-major operands and dimensions aligned to 16, and is unavailable before Hopper. section 8.2.4 section 8.2.4

