liveBook · Manning

Double quantization

As explained in Transformers in Action

Double quantization is a quantization technique that quantizes already-quantized values or their associated quantization parameters to reduce memory use.

Double quantization reduces memory use by applying a second quantization layer to the constants produced during an initial quantization step. In QLoRA, this lowers the storage overhead of quantized model weights while supporting memory-efficient fine-tuning. section 8.3.4 section 8.3

The technique fits into a quantized fine-tuning workflow as follows:

  • Convert pretrained weights to a low-bit representation, such as NF4. section 8.3.4
  • Quantize the constants used by that first quantization step to reduce their additional memory demand. section 8.3.4
  • Use the result with LoRA in QLoRA to fine-tune large language models under tighter GPU-memory limits. section 8.3

Enabling double quantization with bitsandbytes

Set `load_in_4bit=True` and `bnb_4bit_use_double_quant=True` in `BitsAndBytesConfig`. The same configuration selects `bnb_4bit_quant_type="nf4"` and uses `torch.bfloat16` for computation while storing weights in the quantized format. section 8.3.4

How Transformers in Action covers it

How deep it goes
Explains the second quantization layer and its memory rationale but does not detail the underlying quantization algorithm.
Examples
Demonstrates `bnb_4bit_use_double_quant=True` in a bitsandbytes configuration for loading Falcon 7B.
What you can do after
Apply it via BitsAndBytesConfig when loading a 4-bit model for QLoRA fine-tuning.
How it connects
Connects quantization constants with QLoRA, LoRA, NF4, and low-rank adaptation.
Cover of Transformers in Action
liveBookmanning.com
Read Transformers in Action on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.