liveBook · Manning

Quantization

As explained in Building Specialized AI Systems

Quantization is the representation of model values with lower numerical precision to reduce memory use and computation.

Quantization represents model values with fewer bits to reduce memory use and, in some cases, computation. A practical fine-tuning setup can load a base model at 4-bit precision rather than the standard 32-bit representation, allowing a capable model to fit consumer hardware while retaining useful performance. section 2.1.2

In parameter-efficient fine-tuning (PEFT), quantization complements targeted updates rather than replacing them. It reduces the footprint of the frozen model while lightweight trainable components adapt it to a specialized task, making large-model adaptation more accessible and deployable under resource constraints. section 3.5 section 1.4.2

Use QLoRA to separate what is compressed from what remains precise:

  • Store the frozen base-model weights at 4-bit precision, reducing their memory requirement fourfold relative to 16-bit storage. section 3.5.2
  • Keep LoRA adapter matrices at 16-bit precision for training stability, while optimizer states remain 32-bit for trainable parameters. section 3.5.2
  • When GPU memory is constrained, offload optimizer states and related components to CPU memory, then page them back to the GPU. section 3.5.2

4-bit QLoRA sentiment classifier

Configure 4-bit loading, double quantization, NF4 quantization type, and bfloat16 compute; load bert-base-uncased with that configuration; then prepare it for k-bit training, enable gradient checkpointing, and attach a LoRA configuration for Rotten Tomatoes classification. section 3.5.2

Quantization is a resource-performance trade-off, not a guarantee of unchanged quality: compression alone can degrade performance. QLoRA addresses that risk by pairing a quantized frozen base model with low-rank updates, aiming to preserve effective fine-tuning while lowering memory and compute demands. section 3.5.2 section 3.8

Figure 3.12 compares LoRA and QLoRA side by side. QLoRA uses a quantized model instead of the original model. It can also offload optimizer states to CPU memory when GPU memory is tight.

Compare LoRA and QLoRA storage choices: the adapter and optimizer precision stay the same, while QLoRA compresses the frozen base model and can use CPU paging. section 3.5.2

How Building Specialized AI Systems covers it

How deep it goes
Explains 4-bit base weights, 16-bit adapters, 32-bit optimizer states, CPU paging, and a concrete configuration.
Examples
Uses a 4-bit LLaMA-3 financial adaptation and a bert-base-uncased Rotten Tomatoes classifier.
What it assumes
Assumes a fine-tuning workflow and introduces quantization as lower-precision storage for model values.
What you can do after
Apply it via BitsAndBytesConfig to load a model in 4-bit precision and prepare it for k-bit LoRA training.
How it connects
Connects quantization with PEFT, LoRA, QLoRA, gradient checkpointing, and optimizer states.

Also explained in

Compare how 22 books explain quantization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.