liveBook · Manning

E4M3

As explained in Quantization and Fast Inference

An 8-bit floating-point format with one sign bit, four exponent bits, and three mantissa bits.

E4M3 is an 8-bit floating-point format with one sign bit, four exponent bits, and three mantissa bits. It represents 253 finite values spanning ±448. Its narrow range makes scaling essential, while its extra mantissa bit relative to E5M2 gives better precision for weights and activations concentrated near zero. section 8.2 section 8.2.2

Figure 8.3 Near-zero density in [0, 0.5]. E4M3 packs 49 representable values into this interval; E5M2 packs 57 (more exponent range means more subnormal granularity); INT8 symmetric fits only 32. Neural network weight and activation distributions peak sharply near zero, so denser coverage here translates directly to lower quantization error where it matters most.

The near-zero comparison shows why E4M3's non-uniform spacing can reduce quantization error for neural-network values concentrated around zero. section 8.2.1

Configure scaling to keep values in range and limit outlier damage:

  • Scale by the tensor's maximum absolute value before quantization, then multiply by the scale after dequantization; otherwise values beyond ±448 clip. section 8.2.2
  • Use block scaling for activations when outliers distort a per-tensor scale. Blocks of 32 or 128 elements localize the effect; block-32 adds 12.5% storage overhead. section 8.2.2
  • Use per-tensor scales for weights as a practical default; production inference runtimes may choose per-tensor or per-token activation scales. section 8.2.2

Weight-only inference

In an OPT-6.7B test, E4M3 weights stored as one byte each reduced model size from 12.40 GB to 6.21 GB. WikiText-2 perplexity remained 16.26, matching the BF16 result. Direct FP8 matmul requires FP8 tensor cores: the cited PyTorch kernel requires row-major operands and dimensions aligned to 16, and is unavailable before Hopper. section 8.2.4 section 8.2.4

How Quantization and Fast Inference covers it

How deep it goes
Defines the bit layout and range, explains scaling granularity, and specifies a matmul kernel's hardware and layout constraints.
Examples
Compares near-zero representable values and measures OPT-6.7B weight-only compression and perplexity.
What you can do after
Apply via vLLM or TensorRT-LLM to deploy FP8 weights, checking that throughput goals have H100-or-later hardware support.
How it connects
Contrasts E5M2 and INT8, and relates E4M3 to quantization, scaling, BF16, and matrix multiplication.

Also explained in

Compare how 2 books explain E4M3 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.