liveBook · Manning

NormalFloat4

As explained in LLM Customization and Fine-Tuning

A four-bit quantization format designed for neural-network weights that are approximately normally distributed.

NormalFloat4 (NF4) is a 4-bit floating-point format tailored to neural-network weights with a normal distribution. It provides higher precision near zero, making it a preferred 4-bit format when accuracy matters. section 5.5

NF4 is used to quantize a base model’s weights in QLoRA; the weights remain unchanged during training while higher-precision LoRA adapter parameters are trained. Double quantization is a separate optimization that compresses quantization scales and offsets. section 5.5 section 5.1.6

Listing 5.5 Loading a base model with 4-bit quantization for QLoRA
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, prepare_model_for_kbit_training
from trl import SFTTrainer
 
# Configure 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",             # NormalFloat4 quantization
    bnb_4bit_use_double_quant=True,  # double quantization, more savings
    bnb_4bit_compute_dtype=torch.bfloat16  # Compute in bfloat16 for speed
)
 
# Load model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto"
)
 
# k-bit training prep: gradient checkpointing on 4-bit weights
model = prepare_model_for_kbit_training(model)
 
# Rest is the same as step 2: LoRA config, load data, train with SFTTrainer
lora_config = LoraConfig(r=8, lora_alpha=16, ...)
trainer = SFTTrainer(model=model, peft_config=lora_config, ...)
trainer.train()

The configuration selects NF4 for 4-bit loading and enables double quantization; it then prepares the model for k-bit training. section 5.1.6

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines NF4’s distribution-specific precision advantage but does not derive its quantization scheme.
Examples
Configures NF4 with bitsandbytes in a 4-bit model-loading example.
What you can do after
Configure NF4 for 4-bit base-model loading in a QLoRA training pipeline.
How it connects
Connects NF4 with quantization, QLoRA, LoRA, and double quantization.

Also explained in

Compare how 6 books explain NormalFloat4 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.