NVFP4 is NVIDIA’s 4-bit floating-point format, designed to reduce model storage while preserving more quality than coarse-grained FP4 scaling. It stores E2M1 values in blocks of 16, with an FP8 E4M3 scale per block and one FP32 scale for the tensor. section 8.3.4
Hardware determines whether NVFP4 saves inference time as well as memory:
- Blackwell FP4 tensor cores consume NVFP4 natively and deliver twice the TFLOPS of FP8 on the same chip. section 8.3.4
- On H100 and earlier GPUs, FP4 weights must be dequantized before matrix multiplication, so FP4 offers storage savings but no throughput advantage over storing weights in FP8. section 8.3.4
- NVIDIA ModelOpt produces NVFP4 checkpoints; TensorRT-LLM and vLLM support serving them. PyTorch has no native FP4 dtype at the time described. section 8.3.4
OPT-6.7B storage and quality
NVFP4 block-16 is estimated at 3.50 GB, versus 4.66 GB for block-16 FP4 with FP32 scales. In the reported comparison, per-tensor FP4 shrinks the model to 3.11 GB but raises perplexity from 16.26 to 26.09; block scaling largely avoids that quality cliff. section 8.3.4
Figure 8.11 FP4 inference: memory and quality on OPT-6.7B. Left: model size in GB. Right: WikiText-2 perplexity. Per-tensor FP4 (cross-hatched, second bar from left) compresses to 3.11 GB but perplexity spikes to 26.09—a 60% degradation. Block-16 FP4 with FP32 scales (solid fill) recovers to 16.49 (+1.4%) at 4.66 GB. Block-32 FP4 (diagonal hatch) achieves 16.54 (+1.7%) at 3.89 GB. The BF16 baseline (dotted fill) is 12.40 GB at 16.26 perplexity.

Compare FP4 configurations by model size and WikiText-2 perplexity. section 8.3.4
