How deeply does Transformers in Action cover NormalFloat4?

Explains NF4’s distributional rationale, quantile quantization, double quantization, and storage-computation type separation.

NormalFloat4

As explained in Transformers in Action

NF4 is a four-bit numerical data type optimized for representing neural-network weights with an approximately normal distribution.

NormalFloat4 (NF4) is a four-bit numerical data type designed for neural-network weights that approximately follow a normal (Gaussian) distribution. In QLoRA, it reduces storage requirements while supporting memory-efficient fine-tuning. section 8.3.4 section 8.3

NF4’s implementation role combines several memory-saving choices:

Packs normally distributed weights into four bits, assigning denser representation to more common values according to information-theoretic principles. section 8.3.4
Uses quantile quantization to approximate input-tensor quantiles through the empirical cumulative distribution function. section 8.3.4
Can be paired with double quantization, which quantizes the constants from the first quantization step to reduce additional memory use. section 8.3.4
Serves as the storage type while QLoRA uses bfloat16 as the computation type. section 8.3.4

Configure NF4 with bitsandbytes

A four-bit model configuration enables NF4 storage, double quantization, and bfloat16 computation: `BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16)`. Passing this configuration to `AutoModelForCausalLM.from_pretrained` loads the quantized model.

How Transformers in Action covers it

liveBook manning.com

Read Transformers in Action on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Building Reliable AI SystemsNF4 is a four-bit quantization data type with quantization bins distributed according to a normal distribution.

Compare how 2 books explain NormalFloat4 →

Related concepts

QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that quantizes a pretrained model’s weights while training low-rank adapters to reduce memory requirements.
uniform quantizationcontrasts with · Uniform quantization maps numerical values to discrete levels separated by a constant step size across a specified range.
double quantizationuses · Double quantization is a technique that quantizes parameters, such as scale factors, of an already quantized representation, often to reduce storage or memory overhead.
int4 quantizationalternative to · INT4 quantization is a technique that represents model weights, activations, or computations using four-bit integer values to reduce storage and computational requirements.
weightuses · A weight is a numerical quantity assigned to an element to represent its contribution, influence, importance, strength, or resource amount in a calculation or system.
memory usageaddresses · Memory usage is the amount of computer memory occupied or allocated by a program, process, data, or computation during operation.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.