liveBook · Manning

Low-rank adaptation

As explained in LLM Customization and Fine-Tuning

A learned, low-rank additive change to a model's weights.

When prompting falls short but full fine-tuning is too costly, low-rank adaptation (LoRA) offers a smaller weight update. It freezes the pretrained model and trains task-specific adapter matrices, which are combined with the base model at inference. section 5.2 section 5.2.1

For a square weight matrix W of size d × d, LoRA learns two matrices A and B of sizes d × r and r × d. Their product is the additive update ΔW = A × B; the effective weight is W + ΔW. The adapter can be used or removed without changing the base weights. section 5.2.1

Training only the adapter reduces the number of trainable parameters and can make fine-tuning feasible on smaller GPUs. Adapters can also be swapped for different tasks on the same base model. Inference is not necessarily faster: an additional matrix multiplication can increase latency on some hardware. section 5.2 section 5.2.2

Choose and evaluate LoRA as part of an adaptation workflow:

  • Establish a prompting baseline on held-out examples before training. If you lack a few hundred high-quality, representative training examples, curate data first. section 1.7
  • Prepare message-format data, load the base model, configure and train the adapter, then save it and compare its outputs against the baseline. Hugging Face TRL's SFTTrainer handles adapter wrapping, tokenization, and prompt masking from a LoRA configuration. section 2.8 section 5.1
  • Evaluate task quality and test for safety regressions using known harmful prompts. Include explicit refusal examples in training data. section 5.13
Listing 2.3 Step 2, load Qwen3-4B-Instruct-2507 and configure a LoRA adapter
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig
 
BASE_MODEL = "Qwen/Qwen3-4B-Instruct-2507"
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL, trust_remote_code=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
 
model = AutoModelForCausalLM.from_pretrained(
    BASE_MODEL, device_map="auto", dtype="auto", trust_remote_code=True,
)
model.gradient_checkpointing_enable()
model.config.use_cache = False
 
lora_config = LoraConfig(
    r=16, lora_alpha=32, lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "up_proj", "gate_proj", "down_proj"],
    bias="none", task_type="CAUSAL_LM",
)

A configuration for Qwen3-4B shows rank, alpha, dropout, and the attention and feedforward modules targeted by an adapter. section 2.8

IT support house style

An IT support adapter was trained on 166 examples in a fixed house format and compared with minimally prompted and strongly format-prompted base models on six held-out questions. In that small test, only the adapter reached the house format and named the internal tools; the format prompt did not include the internal glossary. section 5.1.8

Use prompting or a decode-time schema when they meet a formatting requirement without training. Use retrieval-augmented generation when knowledge changes frequently or answers need verifiable citations; it can also complement LoRA. If a well-tested adapter plateaus below the quality target, consider full supervised fine-tuning, provided the data and maintenance budget justify it. section 1.6.2 section 5.13 section 6.13

For memory-constrained hardware, QLoRA combines LoRA with a 4-bit base model. LoRA's low-rank constraint also limits capacity: larger datasets or complex tasks may warrant testing full fine-tuning rather than assuming the adapter will close the remaining gap. section 5.13 section 5.2.2

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains frozen weights and low-rank matrix updates, then walks through adapter training, evaluation, safety checks, and deployment tradeoffs.
Examples
Compares an IT support adapter with prompted baselines on six held-out questions and trains a Qwen3-4B adapter in a single-GPU quickstart.
What it assumes
Requires curated task examples and a held-out baseline for a practical training decision, while introducing adapter mechanics and the training workflow.
What you can do after
Apply it via Hugging Face PEFT and TRL to train, save, and evaluate an adapter, including checks for safety regressions.
How it connects
Connects LoRA to parameter-efficient fine-tuning, QLoRA, supervised fine-tuning, retrieval-augmented generation, and knowledge distillation.

Also explained in

Compare how 17 books explain low-rank adaptation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.