liveBook · Manning

NormalFloat4

As explained in Rearchitecting LLMs

4-bit NormalFloat quantization represents model weights with a four-bit NormalFloat numerical encoding tailored to the distribution of language-model weights.

NF4 is a four-bit format for pretrained language-model weights. Its 16 representable values are concentrated near zero, where normally distributed weights are densest, reducing the distortion of uniform four-bit quantization. section 7.2

NF4 is a storage format, not a compute format: weights are dequantized to 16-bit precision for operations. Its main fine-tuning benefit is reducing base-model memory, leaving more GPU space for gradients and optimizer states. Quantization loses precision, and if memory is not the bottleneck, training without it is faster. section 7.2 section 7.6

Figure 7.3 Weight distribution after quantization and dequantization across three methods. 8-bit uniform (left) closely matches the original Gaussian. 4-bit uniform (center) introduces severe distortion, with values collapsing into a small number of discrete levels. NF4 (right), also 4-bit, recovers the distribution more faithfully by concentrating its quantization levels where weights are densest.

Compare how 8-bit uniform, 4-bit uniform, and NF4 quantization preserve the weight distribution. section 7.2

GPU trade-offs

In the reported T4 and L4 tests, NF4 weights increased generation throughput, while FP8 KV-cache quantization offered larger capacity gains. The results vary by hardware and optimization goal. section 8.2.2 section 8.2.2

How Rearchitecting LLMs covers it

How deep it goes
Explains level placement, compares uniform quantization, and distinguishes storage from computation without detailing the quantization algorithm.
Examples
Uses weight-distribution comparisons and T4/L4 inference results to illustrate quantization distortion and hardware-dependent throughput.
What you can do after
Operate/configure it by setting bnb_4bit_quant_type="nf4" for QLoRA fine-tuning.
How it connects
Connects NF4 to quantization, QLoRA, parameter-efficient fine-tuning, and KV-cache alternatives.

Also explained in

Compare how 6 books explain NormalFloat4 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.