liveBook · Manning

FP4

As explained in Quantization and Fast Inference

A family of 4-bit floating-point formats for reduced-precision numerical representation.

FP4 stores each weight in 4 bits, halving the weight storage of 8-bit formats. Its limited representable values make scaling granularity critical: per-tensor scaling can save memory but severely damage model quality. section 8.3 section 8.6

E2M1 uses one sign bit, two exponent bits, and one mantissa bit with exponent bias 1. It reserves no bit patterns for infinity or NaN; its positive values are 0, 0.5, 1, 1.5, 2, 3, 4, and 6. Unlike uniformly spaced INT4, FP4 places more values near zero. section 8.3.1 section 8.3.1

Choose scaling and hardware together:

  • Prefer blockwise scaling over per-tensor scaling. On OPT-6.7B, block-16 FP32 scales brought perplexity to 16.49 from a BF16 baseline of 16.26; per-tensor FP4 reached 26.09. section 8.3.4
  • On Blackwell, NVFP4 uses FP8 scales per 16-value block plus a tensor-level FP32 scale; MXFP4 uses power-of-two scales per 32-value block. Earlier GPUs require dequantization before matmul, so FP4 offers no throughput advantage over FP8 there. section 8.3.4
Table 8.2 Complete FP4 E2M1 positive value table
Bits
Layout (S_EE_M)
Exponent field
Mantissa field
Type
Value
0000
0_00_0
0
0
Zero
0.0
0001
0_00_1
0
1
Subnormal
0.5
0010
0_01_0
1
0
Normal
1.0
0011
0_01_1
1
1
Normal
1.5
0100
0_10_0
2
0
Normal
2.0
0101
0_10_1
2
1
Normal
3.0
0110
0_11_0
3
0
Normal
4.0
0111
0_11_1
3
1
Normal
6.0

Inspect the complete E2M1 positive-value table, including bit patterns and normal versus subnormal values. section 8.3.1

How Quantization and Fast Inference covers it

How deep it goes
Defines E2M1 encoding and compares scaling configurations with measured model quality and memory use.
Examples
Compares per-tensor, block-16, and block-32 FP4 inference on OPT-6.7B using WikiText-2 perplexity and estimated model size.
What you can do after
Apply blockwise scaling and match FP4 deployment to GPU support to balance model quality, storage, and throughput.
How it connects
Contrasts FP4 with INT4 and FP8, and connects it to blockwise scaling, NVFP4, and MXFP4.

Also explained in

Compare how 2 books explain FP4 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.