Double quantization reduces the memory overhead of blockwise quantization by quantizing its scale factors. NF4 stores one FP32 scale per 64 weights; double quantization converts those scales to FP8 and stores one FP32 super-scale per 256 blocks. section 8.4
Scale-storage savings
For Llama-2-7B’s 6.6 billion linear parameters, blocks of 64 require about 103 million scales. Storing each as FP32 takes roughly 390 MB; double quantization reduces this to about 105 MB, saving 0.29 GB. section 8.4
Listing 8.1 Loading Llama-2-7B in NF4 with QLoRA adapters
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4", #A
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True, #B
)
model = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-2-7b-hf",
quantization_config=bnb_config,
device_map="auto",
)
lora_config = LoraConfig(
r=16, lora_alpha=32,
target_modules=["q_proj", "k_proj", "v_proj", #C
"o_proj", "gate_proj",
"up_proj", "down_proj"],
lora_dropout=0.05, bias="none",
task_type="CAUSAL_LM",
)
model = get_peft_model(
prepare_model_for_kbit_training(model), lora_config
)
The QLoRA configuration enables double quantization with `bnb_4bit_use_double_quant=True`. section 8.4


