liveBook · Manning

Double quantization

As explained in Quantization and Fast Inference

Double quantization quantizes quantization parameters, such as block scales, to reduce their storage cost.

Double quantization reduces the memory overhead of blockwise quantization by quantizing its scale factors. NF4 stores one FP32 scale per 64 weights; double quantization converts those scales to FP8 and stores one FP32 super-scale per 256 blocks. section 8.4

Scale-storage savings

For Llama-2-7B’s 6.6 billion linear parameters, blocks of 64 require about 103 million scales. Storing each as FP32 takes roughly 390 MB; double quantization reduces this to about 105 MB, saving 0.29 GB. section 8.4

Listing 8.1 Loading Llama-2-7B in NF4 with QLoRA adapters
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",                        #A
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,                    #B
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)
 
lora_config = LoraConfig(
    r=16, lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj",     #C
                     "o_proj", "gate_proj",
                     "up_proj", "down_proj"],
    lora_dropout=0.05, bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(
    prepare_model_for_kbit_training(model), lora_config
)
 

The QLoRA configuration enables double quantization with `bnb_4bit_use_double_quant=True`. section 8.4

How Quantization and Fast Inference covers it

How deep it goes
Quantizes FP32 block scales to FP8 and stores one FP32 super-scale per 256 blocks.
Examples
Calculates scale-storage costs for Llama-2-7B, comparing roughly 390 MB with 105 MB.
What you can do after
Configure bitsandbytes to apply double quantization with `bnb_4bit_use_double_quant=True`.
How it connects
Connects NF4 and blockwise absmax scaling with QLoRA fine-tuning.

Also explained in

Compare how 3 books explain double quantization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.