liveBook · Manning

QLoRA

As explained in Building Specialized AI Systems

QLoRA is a parameter-efficient fine-tuning method that combines low-rank adaptation with quantization of the base model.

QLoRA (Quantized LoRA) is a parameter-efficient fine-tuning technique that keeps a pretrained base model frozen, stores it in 4-bit form, and trains 16-bit LoRA adapter matrices. It reduces the memory burden of specialization while retaining low-rank adaptation, making large-model fine-tuning more practical on constrained hardware. section 3.5.2 section 1.4.2

The central trade-off is compression versus effective adaptation. Quantization alone can reduce performance, so QLoRA combines compressed frozen weights with trainable low-rank updates. Optimizer states remain 32-bit for trainable parameters; when GPU memory is tight, CPU paging can offload states and components temporarily. section 3.5.2

A practical QLoRA workflow is:

  • Load a pretrained classification model with a 4-bit quantization configuration; the example uses double quantization, NF4, bfloat16 computation, and automatic device placement. section 3.5.2
  • Prepare the model for k-bit training and enable gradient checkpointing to further manage memory. section 3.5.2
  • Tokenize and split labeled data, then attach a LoRA configuration targeting selected modules while retaining the task classifier. section 3.5.2
  • Train with evaluation and checkpointing settings, using a paged 8-bit AdamW optimizer in the illustrated configuration. section 3.5.2

Sentiment classification with a quantized base model

A Rotten Tomatoes dataset is mapped to NEGATIVE and POSITIVE labels, tokenized to a maximum length of 512, and used to fine-tune bert-base-uncased. The base model is loaded in 4-bit form; LoRA targets query and value modules with rank 16, while the classifier is retained for sequence classification. section 3.5.2

Figure 3.12 compares LoRA and QLoRA side by side. QLoRA uses a quantized model instead of the original model. It can also offload optimizer states to CPU memory when GPU memory is tight.

Compare the storage layout directly: LoRA retains a 16-bit frozen base model, whereas QLoRA uses a 4-bit base model while keeping 16-bit adapters and 32-bit optimizer states. section 3.5.2

How Building Specialized AI Systems covers it

How deep it goes
Explains the 4-bit base-model, 16-bit adapter, optimizer-state, CPU-paging, and training-configuration choices through a worked implementation.
Examples
Uses Rotten Tomatoes sentiment classification with bert-base-uncased, labeled outputs, and query/value LoRA adapters.
What it assumes
Builds from pretrained weights and introduces QLoRA as an adaptation method rather than training a model from scratch.
What you can do after
Apply it via Transformers and PEFT to configure, prepare, and train a 4-bit quantized model with LoRA adapters.
How it connects
Connects quantization, LoRA, parameter-efficient fine-tuning, gradient checkpointing, and catastrophic forgetting.

Also explained in

Compare how 7 books explain QLoRA →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Building Specialized AI Systems. The section links point to the parts of the book it draws on.