liveBook · Manning

NVFP4

As explained in Quantization and Fast Inference

NVFP4 is NVIDIA's 4-bit floating-point format that stores block scales in FP8 and a tensor-wide scale in FP32.

NVFP4 is NVIDIA’s 4-bit floating-point format, designed to reduce model storage while preserving more quality than coarse-grained FP4 scaling. It stores E2M1 values in blocks of 16, with an FP8 E4M3 scale per block and one FP32 scale for the tensor. section 8.3.4

Hardware determines whether NVFP4 saves inference time as well as memory:

  • Blackwell FP4 tensor cores consume NVFP4 natively and deliver twice the TFLOPS of FP8 on the same chip. section 8.3.4
  • On H100 and earlier GPUs, FP4 weights must be dequantized before matrix multiplication, so FP4 offers storage savings but no throughput advantage over storing weights in FP8. section 8.3.4
  • NVIDIA ModelOpt produces NVFP4 checkpoints; TensorRT-LLM and vLLM support serving them. PyTorch has no native FP4 dtype at the time described. section 8.3.4

OPT-6.7B storage and quality

NVFP4 block-16 is estimated at 3.50 GB, versus 4.66 GB for block-16 FP4 with FP32 scales. In the reported comparison, per-tensor FP4 shrinks the model to 3.11 GB but raises perplexity from 16.26 to 26.09; block scaling largely avoids that quality cliff. section 8.3.4

Figure 8.11 FP4 inference: memory and quality on OPT-6.7B. Left: model size in GB. Right: WikiText-2 perplexity. Per-tensor FP4 (cross-hatched, second bar from left) compresses to 3.11 GB but perplexity spikes to 26.09—a 60% degradation. Block-16 FP4 with FP32 scales (solid fill) recovers to 16.49 (+1.4%) at 4.66 GB. Block-32 FP4 (diagonal hatch) achieves 16.54 (+1.7%) at 3.89 GB. The BF16 baseline (dotted fill) is 12.40 GB at 16.26 perplexity.

Compare FP4 configurations by model size and WikiText-2 perplexity. section 8.3.4

How Quantization and Fast Inference covers it

How deep it goes
Explains E2M1 values, FP8 block scales, the tensor-wide FP32 scale, and hardware constraints.
Examples
Compares OPT-6.7B model size and perplexity across per-tensor, block-scaled, and NVFP4 configurations.
What you can do after
Operate/configure deployment by matching NVFP4 checkpoints and inference expectations to the available GPU.
How it connects
Connects FP4 scaling to MXFP4, FP8, E2M1, and Blackwell tensor cores.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.