liveBook · Manning

Quantization

As explained in LLM Customization and Fine-Tuning

Representing model values at reduced numerical precision to lower storage or computational requirements.

Quantization reduces the precision of model weights to lower memory requirements. Storing weights in 4 bits rather than 16 cuts weight storage by roughly fourfold, with a possible accuracy tradeoff. This can make fine-tuning feasible when a model otherwise exceeds available GPU memory. section 5.5

QLoRA combines 4-bit quantization with LoRA: the quantized base weights remain frozen, while higher-precision adapter weights are trained. Two techniques improve memory efficiency: NormalFloat4 (NF4) allocates more precision near zero for normally distributed weights, and double quantization compresses quantization scale and offset values, saving about 0.4 bits per parameter. section 5.5

Listing 5.5 Loading a base model with 4-bit quantization for QLoRA
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, prepare_model_for_kbit_training
from trl import SFTTrainer
 
# Configure 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",             # NormalFloat4 quantization
    bnb_4bit_use_double_quant=True,  # double quantization, more savings
    bnb_4bit_compute_dtype=torch.bfloat16  # Compute in bfloat16 for speed
)
 
# Load model in 4-bit
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto"
)
 
# k-bit training prep: gradient checkpointing on 4-bit weights
model = prepare_model_for_kbit_training(model)
 
# Rest is the same as step 2: LoRA config, load data, train with SFTTrainer
lora_config = LoraConfig(r=8, lora_alpha=16, ...)
trainer = SFTTrainer(model=model, peft_config=lora_config, ...)
trainer.train()

This configuration loads a base model in 4-bit NF4 with double quantization, uses bfloat16 for computation, and prepares the model for k-bit training before applying LoRA. section 5.1.6

Trading training time for GPU memory

For Qwen3-4B, QLoRA uses approximately 4–6 GB of GPU memory, compared with 8–12 GB for regular LoRA. On the same 24 GB GPU, a run over 450 examples for three epochs took 14–15 minutes with QLoRA versus 11 minutes with LoRA; quantization and dequantization add overhead. section 5.1.6

Table 5.5 Typical memory requirements for fine-tuning a model of various sizes
Model Size
Full Fine-Tuning (VRAM)
LoRA (VRAM)
QLoRA (VRAM)
Notes
7 B
~60 to 67 GB
16 to 20 GB
8 to 10 GB
Fits on a single 24 GB GPU using LoRA. QLoRA enables larger batch sizes or longer contexts.
13 B
~120 to 130 GB
30 to 36 GB
10 to 18 GB
LoRA often requires two 24 GB GPUs for 13B; QLoRA may fit on one high-end consumer GPU.
65 B
600 GB+ (multi-GPU)
~146 GB
~48 GB+
65B LoRA needs a multi-GPU node (e.g., 8x H100). QLoRA fits 65B on a single 48 GB GPU, the QLoRA paper’s headline result [2]; an 80 GB H100 or H200 gives headroom for longer contexts.

Compare typical VRAM requirements across full fine-tuning, LoRA, and QLoRA for 7B, 13B, and 65B models. Requirements vary with the model and workload. section 5.5

Choose regular LoRA when it fits the available hardware; use QLoRA when memory constraints justify slower training. Training-time bitsandbytes quantization differs from serving optimization: a usual deployment path merges the adapter into the full-precision base, then quantizes the merged model. Formats such as GPTQ or AWQ typically offer higher inference throughput. section 5.1.6 section 5.5

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains NF4 and double quantization and demonstrates training configuration, while deferring deployment quantization.
Examples
Compares Qwen3-4B training memory and runtime and provides VRAM estimates for 7B, 13B, and 65B models.
What you can do after
Apply it via bitsandbytes to load and prepare a 4-bit model for QLoRA training.
How it connects
Connects quantization with LoRA, parameter-efficient fine-tuning, and inference optimization.

Also explained in

Compare how 22 books explain quantization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.