liveBook · Manning

Double quantization

Double quantization is a technique that quantizes the parameters, such as scale factors, associated with an earlier quantization step to reduce their storage cost.

How 3 Manning books explain it

Quantization and Fast Inference

Double quantization quantizes quantization parameters, such as block scales, to reduce their storage cost.

How it is used
Reduces scale-storage overhead when using NF4 weights for QLoRA fine-tuning.
How deep it goes
Quantizes FP32 block scales to FP8 and stores one FP32 super-scale per 256 blocks.
Examples
Calculates scale-storage costs for Llama-2-7B, comparing roughly 390 MB with 105 MB.
What you can do after
Configure bitsandbytes to apply double quantization with `bnb_4bit_use_double_quant=True`.

Read how Quantization and Fast Inference explains double quantization →

Building Specialized AI Systems

Double quantization is a two-level quantization method that quantizes model weights and then quantizes the constants used to scale those weights.

How it is used
Reduces memory requirements when fine-tuning quantized models through QLoRA.
How deep it goes
Explains the two quantization levels and provides configuration steps, but does not detail the quantization algorithm mathematically.
Examples
Uses BitsAndBytesConfig with 4-bit loading, NF4, bfloat16 computation, and a Rotten Tomatoes classification dataset.
What you can do after
Apply it via BitsAndBytesConfig when loading a 4-bit model for QLoRA fine-tuning.

Read how Building Specialized AI Systems explains double quantization →

Transformers in Action

Double quantization is a quantization technique that quantizes already-quantized values or their associated quantization parameters to reduce memory use.

How it is used
Reduces quantization overhead during QLoRA fine-tuning of large language models under constrained memory.
How deep it goes
Explains the second quantization layer and its memory rationale but does not detail the underlying quantization algorithm.
Examples
Demonstrates `bnb_4bit_use_double_quant=True` in a bitsandbytes configuration for loading Falcon 7B.
What you can do after
Apply it via BitsAndBytesConfig when loading a 4-bit model for QLoRA fine-tuning.

Read how Transformers in Action explains double quantization →

What to know first

Related concepts

Questions about double quantization

What is double quantization?

Double quantization is a technique that quantizes the parameters, such as scale factors, associated with an earlier quantization step to reduce their storage cost.

Which Manning books explain double quantization?

Quantization and Fast Inference, Building Specialized AI Systems and Transformers in Action.

What should I know before learning double quantization?

Manning books that teach double quantization usually introduce foundation model, low-rank adaptation, bfloat16 and fine-tuning first.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.