liveBook · Manning

FP8

As explained in Quantization and Fast Inference

A family of 8-bit floating-point formats for reduced-precision numerical computation.

FP8 is a family of 8-bit floating-point formats used to reduce model storage and accelerate computation. On H100, FP8 tensor cores have twice the peak throughput of BF16 tensor cores; realized inference gains depend on the workload and an FP8-capable kernel. Without such a kernel, FP8 can still save storage, but dequantization to BF16 removes the FP8 compute advantage. section 8.1 section 8.2.4

The two formats trade precision against range:

  • E4M3 uses one sign bit, four exponent bits, and three mantissa bits. Its finite range is ±448, with eight levels per power-of-two interval. Use it for weights and forward-pass activations, where the extra mantissa bit reduces rounding error. section 8.2
  • E5M2 uses one sign bit, five exponent bits, and two mantissa bits. Its range extends to ±57,344, with four levels per power-of-two interval. Use it for gradients, where range helps accommodate outliers. section 8.2

FP8 values are non-uniformly spaced: they are denser near zero and sparser at larger magnitudes. That can reduce error for tensors concentrated near zero, while uniformly distributed values may favor INT8. Scaling remains essential to fit tensor values into the format’s range; use per-tensor scaling for weights and block scaling for activations. section 8.2.1 section 8.6

Figure 8.3 Near-zero density in [0, 0.5]. E4M3 packs 49 representable values into this interval; E5M2 packs 57 (more exponent range means more subnormal granularity); INT8 symmetric fits only 32. Neural network weight and activation distributions peak sharply near zero, so denser coverage here translates directly to lower quantization error where it matters most.

The near-zero number line compares representable-value density across E4M3, E5M2, and INT8. section 8.2.1

Weight reconstruction versus end-to-end quality

For OPT-6.7B fc1 weight matrices, INT8 per-channel reconstruction MSE averaged approximately 3.5×10⁻⁸, versus 2.1×10⁻⁷ for FP8 E4M3 per-tensor. This is not an equal-granularity comparison: INT8 uses a separate scale per channel, while FP8 uses one per matrix. Reconstruction error alone does not determine end-to-end perplexity. section 8.2.3

Weight-only deployment

Per-tensor E4M3 storage reduced OPT-6.7B from 12.40 GB to 6.21 GB, with WikiText-2 perplexity reported as 16.26 for both BF16 and FP8. This test dequantized weights to BF16 for matmul, so its storage and quality result is not itself an FP8 compute-throughput measurement. section 8.2.4

For direct FP8 matmul in PyTorch, torch._scaled_mm takes two FP8 operands and two FP32 scale factors and returns BF16. It requires row-major layout and dimensions aligned to 16. Check hardware and kernel support before counting on throughput gains; H100 or later is the stated hardware gate. section 8.2.4

How Quantization and Fast Inference covers it

How deep it goes
Explains bit layouts, representable-value distributions, scaling choices, reconstruction error, and FP8 kernel constraints.
Examples
Compares OPT-6.7B weight reconstruction and measures weight-only storage and WikiText-2 perplexity.
What you can do after
Operate/configure FP8 by selecting formats and scaling strategies, then checking hardware and kernel requirements.
How it connects
Connects floating-point quantization with INT8, mixed-precision training, scaling granularity, and tensor cores.

Also explained in

Compare how 2 books explain FP8 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.