liveBook · Manning

NormalFloat4

As explained in Quantization and Fast Inference

NF4 is a 4-bit NormalFloat format that provides nonuniform representable levels suited to normally distributed neural-network weights.

NormalFloat4 (NF4) stores weights in 4 bits using a fixed, nonuniform codebook matched to approximately normally distributed neural-network weights. Its practical use is QLoRA fine-tuning: a frozen NF4 base model saves memory while trainable adapters learn the task. section 8.1 section 8.4

Unlike FP4, NF4 has no exponent or mantissa. Its 16 lookup values are derived from normal-distribution quantiles, with denser spacing near zero and sparser spacing in the tails. The codebook has seven negative values, zero, and eight positive values. INT4 instead uses uniformly spaced levels; both NF4 and INT4 use 4 bits per weight. section 8.1 section 8.4

{-1.00, -0.70, -0.53, -0.39, -0.28, -0.18, -0.09, 0.00,
  0.08,  0.16,  0.25,  0.34,  0.44,  0.56,  0.72,  1.00}

The codebook shows the nonuniform spacing and the asymmetric allocation around zero. section 8.4

Scaling granularity matters alongside the codebook. In a reconstruction comparison on OPT-6.7B fc1 layers, NF4 with blocks of 64 averaged 2.64×10⁻⁶ MSE; INT4 with the same group size had 1.44× higher MSE. FP4 with blocks of 16 was close at 2.65×10⁻⁶, but used four times finer blocks. These results measure weight reconstruction, not end-task quality. section 8.4 section 8.6

Figure 8.14 Weight reconstruction MSE across formats on OPT-6.7B fc1 layers (log scale). NF4 block-64 (shortest solid bars) achieves the lowest 4-bit MSE at 2.64×10⁻⁶ average. INT4 group-64 (diagonal hatch, same block size) is 1.44× worse. INT4 group-128 (GPTQ/AWQ default) is 1.73× worse. FP4 block-16 (backslash hatch) matches NF4 at 2.65×10⁻⁶ but uses 4× finer blocks. FP8 E4M3 per-tensor (vertical hatch, shortest bars) remains 12× better than any 4-bit format.

Compare reconstruction error across formats and scaling granularities. section 8.4

NF4 has no dedicated hardware acceleration. In the described QLoRA path, bitsandbytes CUDA kernels dequantize weights before BF16 matrix multiplication. Choose NF4 for fine-tuning memory pressure rather than assuming a native 4-bit throughput gain. section 8.6

Llama-2-7B QLoRA

Configure bitsandbytes with 4-bit loading, quantization type "nf4", BF16 compute, and double quantization; prepare the model for k-bit training and attach LoRA adapters. On H100, the NF4 base occupied 3.60 GB, and the measured training peak was 12.31 GB with 8-bit optimizer states, gradients, and activations. section 8.4

Listing 8.1 Loading Llama-2-7B in NF4 with QLoRA adapters
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",                        #A
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,                    #B
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)
 
lora_config = LoraConfig(
    r=16, lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj",     #C
                     "o_proj", "gate_proj",
                     "up_proj", "down_proj"],
    lora_dropout=0.05, bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(
    prepare_model_for_kbit_training(model), lora_config
)
 

The configuration specifies NF4 loading, BF16 compute, double quantization, and adapter targets. section 8.4

How Quantization and Fast Inference covers it

How deep it goes
Explains codebook construction, blockwise scaling, reconstruction error, hardware constraints, and a concrete adapter configuration.
Examples
Compares OPT-6.7B weight reconstruction errors and configures Llama-2-7B QLoRA with a measured training-memory breakdown.
What you can do after
Apply NF4 via bitsandbytes and LoRA to configure a memory-efficient fine-tuning workflow.
How it connects
Connects NF4 to QLoRA, LoRA, quantization, blockwise scaling, INT4, and FP4.

Also explained in

Compare how 6 books explain NormalFloat4 →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.