liveBook · Manning

Low-rank adaptation

As explained in Quantization and Fast Inference

LoRA is a parameter-efficient fine-tuning method that freezes pretrained weights and learns low-rank update matrices.

Low-rank adaptation trains small, full-precision adapters while keeping pretrained weights frozen. In QLoRA, those base weights are quantized to 4-bit NF4, reducing the memory needed to fine-tune a model. section 8.4

Listing 8.1 Loading Llama-2-7B in NF4 with QLoRA adapters
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",                        #A
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,                    #B
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=bnb_config,
    device_map="auto",
)
 
lora_config = LoraConfig(
    r=16, lora_alpha=32,
    target_modules=["q_proj", "k_proj", "v_proj",     #C
                     "o_proj", "gate_proj",
                     "up_proj", "down_proj"],
    lora_dropout=0.05, bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(
    prepare_model_for_kbit_training(model), lora_config
)
 

Configuration example for loading a Llama-2-7B base in NF4 and attaching rank-16 LoRA adapters. section 8.4

Training memory

A measured Llama-2-7B QLoRA run uses 12.31 GB peak GPU memory, including optimizer states, gradients, and activations.

How Quantization and Fast Inference covers it

How deep it goes
Explains the frozen-base and trainable-adapter roles, then gives a QLoRA configuration and measured memory costs.
Examples
Uses rank-16 adapters on Llama-2-7B and reports a 12.31 GB peak training footprint.
What you can do after
Apply it via bitsandbytes and PEFT to load an NF4 base and attach trainable adapters.
How it connects
Ties LoRA to QLoRA, NF4 quantization, and pretrained-model fine-tuning.

Also explained in

Compare how 17 books explain low-rank adaptation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.