NormalFloat4 (NF4) is a four-bit quantization format used when GPU memory limits LLM fine-tuning. Its quantization bins follow a normal distribution, giving values near zero—common in neural-network weights—better precision. In QLoRA, NF4 helps retain useful weight information despite reduced storage precision. section 5.3.3 section 5.3.4
NF4 is one part of a memory-efficient training setup:
- Store model weights in four bits rather than 16 bits, reducing weight-storage memory by approximately 75%. section 5.3.4
- Perform computations in float16 even while weights remain stored in four bits. section 5.3.4
- Combine the quantized model with LoRA adapters so training updates only a small fraction of parameters; QLoRA also uses double quantization to reduce storage for quantization settings. section 5.3.3
base_model = "NousResearch/Llama-2-7b-chat-hf" #1
bnb_config = BitsAndBytesConfig( #1
load_in_4bit=True, #1
bnb_4bit_quant_type="nf4", #1
bnb_4bit_compute_dtype=torch.float16, #1
)
model = AutoModelForCausalLM.from_pretrained(
base_model,
quantization_config=bnb_config,
device_map="auto",
trust_remote_code=True, # Required for some models that include custom
code in their repository. Only enable for models from trusted sources.
)
model = prepare_model_for_kbit_training(model)
tokenizer = AutoTokenizer.from_pretrained(base_model)
tokenizer.pad_token = tokenizer.eos_token
The model-loading configuration selects NF4 with `bnb_4bit_quant_type="nf4"`, enables four-bit loading, and sets float16 computation. `prepare_model_for_kbit_training` then prepares the loaded model for fine-tuning. section 5.3.4
Start from full-precision model weights and quantize during loading; do not fine-tune a model already saved in pre-quantized form. Existing rounding errors can compound during training and hurt performance. section 5.3.4
