NormalFloat4
NormalFloat4 (NF4) is a nonuniform four-bit quantization format whose levels are chosen from normal-distribution quantiles to represent approximately normally distributed neural-network weights.
Building Reliable AI Systems
NF4 is a four-bit quantization data type with quantization bins distributed according to a normal distribution.
- How it is used
- Enables memory-efficient QLoRA fine-tuning when limited GPU resources make larger models difficult to train.
- How deep it goes
- Defines NF4 bin placement and shows its model-loading setting, without deriving the quantization mathematics.
- Examples
- Configures a Llama-2-7b customer-support assistant with NF4 through `BitsAndBytesConfig`.
- What you can do after
- Apply it via Transformers and bitsandbytes to load four-bit weights for QLoRA fine-tuning.
Read how Building Reliable AI Systems explains NormalFloat4 →
Transformers in Action
NF4 is a four-bit numerical data type optimized for representing neural-network weights with an approximately normal distribution.
- How it is used
- Supports memory-efficient QLoRA fine-tuning by storing normally distributed model weights in a four-bit format.
- How deep it goes
- Explains NF4’s distributional rationale, quantile quantization, double quantization, and storage-computation type separation.
- Examples
- Uses Falcon 7B loading with bitsandbytes and a configuration enabling NF4, double quantization, and bfloat16 computation.
- What you can do after
- Operate/configure four-bit model loading with bitsandbytes using NF4 storage, double quantization, and bfloat16 computation.
Related concepts
- QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that quantizes a pretrained model’s weights while training low-rank adapters to reduce memory requirements.
- uniform quantizationcontrasts with · Uniform quantization maps numerical values to discrete levels separated by a constant step size across a specified range.
- double quantizationuses · Double quantization is a technique that quantizes parameters, such as scale factors, of an already quantized representation, often to reduce storage or memory overhead.
- int4 quantizationalternative to · INT4 quantization is a technique that represents model weights, activations, or computations using four-bit integer values to reduce storage and computational requirements.
- weightuses · A weight is a numerical quantity assigned to an element to represent its contribution, influence, importance, strength, or resource amount in a calculation or system.
- memory usageaddresses · Memory usage is the amount of computer memory occupied or allocated by a program, process, data, or computation during operation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.