How deeply does Transformers in Action cover QLoRA?

Explains NF4, double quantization, computation precision, paged optimizers, and model-loading configuration, stopping short of adapter-training code.

QLoRA

As explained in Transformers in Action

QLoRA is a parameter-efficient fine-tuning method that combines low-rank adapter updates with quantization of the base model.

QLoRA adapts large language models without updating every pretrained parameter. It combines LoRA’s low-rank updates with quantized base-model weights, reducing memory demands while preserving the precision benefits of fine-tuning. section 8.3 section 8.3.4

QLoRA’s implementation combines several memory-saving mechanisms:

Stores weights with 4-bit NormalFloat (NF4), designed for weights following a normal distribution. section 8.3.4
Applies double quantization to the constants used by the first quantization step, reducing additional memory overhead. section 8.3.4
Uses bfloat16 for computation while retaining NF4 for weight storage. section 8.3.4
Uses paged optimizers to manage memory movement between CPU and GPU during gradient-checkpointing spikes. section 8.3.4

Loading a quantized Falcon model

A bitsandbytes configuration enables 4-bit loading, double quantization, NF4 storage, and bfloat16 computation before loading tiiuae/falcon-7b with the quantization configuration. The workflow also measures initial, final, and peak GPU memory. section 8.3.4

How Transformers in Action covers it

liveBook manning.com

Read Transformers in Action on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Building Reliable AI SystemsQuantized Low-Rank Adaptation combines low-rank fine-tuning with quantized model weights to reduce memory requirements.
liveBook manning.com

Generative AI in ActionQLoRA is a low-rank adaptation technique that quantizes adapter parameters to reduce the memory required for fine-tuning large language models.

Compare how 3 books explain QLoRA →

Related concepts

quantizationuses · Quantization maps numerical values to a finite set of discrete representations, often using reduced precision to lower model storage or computation requirements.
NormalFloat4uses · NormalFloat4 (NF4) is a nonuniform four-bit quantization format whose levels are chosen from normal-distribution quantiles to represent approximately normally distributed neural-network weights.
low-rank adaptationuses · Low-rank adaptation is a parameter-efficient fine-tuning method that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
double quantizationuses · Double quantization is a technique that quantizes parameters, such as scale factors, of an already quantized representation, often to reduce storage or memory overhead.
bitsandbytesuses · bitsandbytes is an open-source machine-learning library that provides low-bit optimizers and quantization operations for memory-efficient model computation.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.