Double quantization
Double quantization is a technique that quantizes the parameters, such as scale factors, associated with an earlier quantization step to reduce their storage cost.
Quantization and Fast Inference
Double quantization quantizes quantization parameters, such as block scales, to reduce their storage cost.
- How it is used
- Reduces scale-storage overhead when using NF4 weights for QLoRA fine-tuning.
- How deep it goes
- Quantizes FP32 block scales to FP8 and stores one FP32 super-scale per 256 blocks.
- Examples
- Calculates scale-storage costs for Llama-2-7B, comparing roughly 390 MB with 105 MB.
- What you can do after
- Configure bitsandbytes to apply double quantization with `bnb_4bit_use_double_quant=True`.
Read how Quantization and Fast Inference explains double quantization →
Building Specialized AI Systems
Double quantization is a two-level quantization method that quantizes model weights and then quantizes the constants used to scale those weights.
- How it is used
- Reduces memory requirements when fine-tuning quantized models through QLoRA.
- How deep it goes
- Explains the two quantization levels and provides configuration steps, but does not detail the quantization algorithm mathematically.
- Examples
- Uses BitsAndBytesConfig with 4-bit loading, NF4, bfloat16 computation, and a Rotten Tomatoes classification dataset.
- What you can do after
- Apply it via BitsAndBytesConfig when loading a 4-bit model for QLoRA fine-tuning.
Read how Building Specialized AI Systems explains double quantization →
Transformers in Action
Double quantization is a quantization technique that quantizes already-quantized values or their associated quantization parameters to reduce memory use.
- How it is used
- Reduces quantization overhead during QLoRA fine-tuning of large language models under constrained memory.
- How deep it goes
- Explains the second quantization layer and its memory rationale but does not detail the underlying quantization algorithm.
- Examples
- Demonstrates `bnb_4bit_use_double_quant=True` in a bitsandbytes configuration for loading Falcon 7B.
- What you can do after
- Apply it via BitsAndBytesConfig when loading a 4-bit model for QLoRA fine-tuning.
Read how Transformers in Action explains double quantization →
What to know first
- foundation modelA foundation model is a broadly trained machine-learning model whose learned capabilities can be reused or adapted for a range of downstream tasks.
- low-rank adaptationLow-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
- bfloat16bfloat16 is a 16-bit floating-point format with one sign bit, eight exponent bits, and seven fraction bits, providing a wide numerical range with reduced precision.
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
Related concepts
- QLoRAused by · QLoRA is a parameter-efficient fine-tuning method that combines a quantized pretrained model with trainable low-rank adapters.
- quantization scaleuses · A quantization scale is a multiplicative factor relating quantized values or steps to the numerical values they represent.
Questions about double quantization
What is double quantization?
Double quantization is a technique that quantizes the parameters, such as scale factors, associated with an earlier quantization step to reduce their storage cost.
Which Manning books explain double quantization?
Quantization and Fast Inference, Building Specialized AI Systems and Transformers in Action.
What should I know before learning double quantization?
Manning books that teach double quantization usually introduce foundation model, low-rank adaptation, bfloat16 and fine-tuning first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


