liveBook · Manning

Quantized model

As explained in Rearchitecting LLMs

A quantized model is a neural model whose parameters or computational values are represented using reduced-precision numerical formats.

A model loaded in 4-bit cannot compute gradients directly through its weights. Prepare it with PEFT’s `prepare_model_for_kbit_training` before QLoRA adapter training; the base model stays frozen while adapters train. section 7.3.3

Listing 7.9 Preparing the quantized model to receive gradients
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
 
model = prepare_model_for_kbit_training(model)

See the PEFT call used to prepare a quantized model for gradient-based training. section 7.3.3

How Rearchitecting LLMs covers it

How deep it goes
Explains gradient preparation and frozen base weights, but not the quantization mechanism itself.
Examples
Trains on clinical notes paired with validated JSON extraction targets.
What it assumes
Assumes a model is already loaded in 4-bit and introduces PEFT preparation for training.
What you can do after
Apply Hugging Face PEFT’s `prepare_model_for_kbit_training` before configuring and training adapters.
How it connects
Connects quantized models to QLoRA, LoRA adapters, and PEFT.

Also explained in

Compare how 2 books explain quantized model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.