NormalFloat4 (NF4) is a 4-bit floating-point format tailored to neural-network weights with a normal distribution. It provides higher precision near zero, making it a preferred 4-bit format when accuracy matters. section 5.5
NF4 is used to quantize a base model’s weights in QLoRA; the weights remain unchanged during training while higher-precision LoRA adapter parameters are trained. Double quantization is a separate optimization that compresses quantization scales and offsets. section 5.5 section 5.1.6
Listing 5.5 Loading a base model with 4-bit quantization for QLoRA
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, prepare_model_for_kbit_training
from trl import SFTTrainer
# Configure 4-bit quantization
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4", # NormalFloat4 quantization
bnb_4bit_use_double_quant=True, # double quantization, more savings
bnb_4bit_compute_dtype=torch.bfloat16 # Compute in bfloat16 for speed
)
# Load model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
model_name,
quantization_config=bnb_config,
device_map="auto"
)
# k-bit training prep: gradient checkpointing on 4-bit weights
model = prepare_model_for_kbit_training(model)
# Rest is the same as step 2: LoRA config, load data, train with SFTTrainer
lora_config = LoraConfig(r=8, lora_alpha=16, ...)
trainer = SFTTrainer(model=model, peft_config=lora_config, ...)
trainer.train()The configuration selects NF4 for 4-bit loading and enables double quantization; it then prepares the model for k-bit training. section 5.1.6





