NF4 is a four-bit format for pretrained language-model weights. Its 16 representable values are concentrated near zero, where normally distributed weights are densest, reducing the distortion of uniform four-bit quantization. section 7.2
NF4 is a storage format, not a compute format: weights are dequantized to 16-bit precision for operations. Its main fine-tuning benefit is reducing base-model memory, leaving more GPU space for gradients and optimizer states. Quantization loses precision, and if memory is not the bottleneck, training without it is faster. section 7.2 section 7.6
Figure 7.3 Weight distribution after quantization and dequantization across three methods. 8-bit uniform (left) closely matches the original Gaussian. 4-bit uniform (center) introduces severe distortion, with values collapsing into a small number of discrete levels. NF4 (right), also 4-bit, recovers the distribution more faithfully by concentrating its quantization levels where weights are densest.
Compare how 8-bit uniform, 4-bit uniform, and NF4 quantization preserve the weight distribution. section 7.2
GPU trade-offs
In the reported T4 and L4 tests, NF4 weights increased generation throughput, while FP8 KV-cache quantization offered larger capacity gains. The results vary by hardware and optimization goal. section 8.2.2 section 8.2.2





