liveBook · Manning

Double quantization

As explained in Building Specialized AI Systems

Double quantization is a two-level quantization method that quantizes model weights and then quantizes the constants used to scale those weights.

Double quantization compresses model weights in two stages: it quantizes weights to 4-bit precision, then quantizes their scaling factors and zero points. section 3.5.2

By compressing quantization constants that would otherwise use higher precision, the technique reduces memory overhead while preserving the quality of weight representations. section 3.5.2

QLoRA sentiment classification

A Rotten Tomatoes classifier loads bert-base-uncased in 4-bit precision with double quantization, applies LoRA, and trains with paged AdamW 8-bit optimization.

How Building Specialized AI Systems covers it

How deep it goes
Explains the two quantization levels and provides configuration steps, but does not detail the quantization algorithm mathematically.
Examples
Uses BitsAndBytesConfig with 4-bit loading, NF4, bfloat16 computation, and a Rotten Tomatoes classification dataset.
What it assumes
Assumes familiarity with model fine-tuning, LoRA, and quantized training terminology.
What you can do after
Apply it via BitsAndBytesConfig when loading a 4-bit model for QLoRA fine-tuning.
How it connects
Connects quantization constants with 4-bit quantization, NF4, QLoRA, LoRA, and paged optimizers.

Also explained in

Compare how 3 books explain double quantization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.