liveBook · Manning

NormalFloat4

As explained in Building Specialized AI Systems

NF4 is a four-bit numerical data type designed to represent neural-network weights whose values approximately follow a Gaussian distribution.

NormalFloat4 (NF4) is a four-bit data type for quantizing neural-network weights that approximately follow a Gaussian distribution. It places quantization levels by equal-sized distribution bins, concentrating levels where weights are densest and reducing information loss from outliers. section 3.5.2

NF4 supports memory-efficient QLoRA training by:

  • Storing frozen base-model weights at four-bit precision instead of 16-bit precision. section 3.5.2
  • Combining weight quantization with double quantization of scaling factors and zero points. section 3.5.2
  • Leaving trainable LoRA adapters at 16-bit precision while the base model remains quantized. section 3.5.2

NF4 configuration

A Transformers and BitsAndBytes configuration enables four-bit loading, NF4 quantization, double quantization, and bfloat16 computation: load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", and bnb_4bit_compute_dtype=torch.bfloat16.

How Building Specialized AI Systems covers it

How deep it goes
Explains Gaussian-aware quantile placement, memory trade-offs, and configuration, but defers numerical encoding details.
Examples
Uses a BitsAndBytesConfig for NF4 loading, double quantization, and bfloat16 computation.
What it assumes
Assumes familiarity with neural-network weights, model fine-tuning, and LoRA-based adapters.
What you can do after
Apply it via BitsAndBytesConfig when loading a model for four-bit QLoRA training.
How it connects
Connects quantization with QLoRA, LoRA adapters, double quantization, paged optimizers, and k-bit training.

Also explained in

Compare how 6 books explain NormalFloat4 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.