liveBook · Manning

LoRA rank

As explained in LLM Customization and Fine-Tuning

The rank hyperparameter that sets the dimensionality of a LoRA adapter’s low-rank updates.

LoRA rank (r) sets the dimensionality of an adapter’s low-rank updates and controls its expressivity. Higher rank can represent more complex changes, but typically costs more GPU memory and training time. Rank also affects the update’s scaling: effective weight = W + (α/r) × A × B, where α is the scaling factor. section 5.3

Choose rank against the task and validation results:

  • Try r=4 or 8 for relatively simple changes such as terminology, shorter answers, or a fixed format. section 5.3
  • Start at r=16 for a general-purpose baseline. Try r=32 if validation performance plateaus; if it does not improve results, rank may not be the bottleneck. section 5.3
  • Consider a rank increase before full supervised fine-tuning when LoRA is already within roughly five percentage points of the target metric. section 6.6.1
Table 5.4 Impact of LoRA rank on performance and resources on a sample task
Rank (r)
Alpha (α)
Training Time
Val Accuracy
VRAM
Recommendation
4
8
1.2 hr
61.3%
9.2 GB
Too limited for complex tasks
8
16
1.4 hr
64.8%
9.8 GB
Good for simple style/format tasks
16
32
1.8 hr
67.8%
10.4 GB
Sweet spot for most use cases
32
64
2.4 hr
68.1%
11.9 GB
Marginal gain (+0.3pp) for 33% more time
64
128
3.8 hr
68.3%
14.2 GB
Diminishing returns, approaching full SFT cost
Full SFT
N/A
18+ hr
69.1%
32+ GB
Only +1.3pp over r=16 at ~10x cost; not worth it

On a sample 4B-model task, higher ranks add cost for diminishing validation gains. These figures are a comparison for that task, not guarantees for another workload. section 5.3

A smaller rank did not restore safety

An IT-support adapter trained on 450 helpful examples with no refusal examples reduced the safety refusal rate from 100% to 60%. Retraining at r=8 instead of r=16 left refusal at 60%. Token-F1 was 0.167 versus 0.154, but the difference was within noise on the 50-example test split. Reducing rank did not fix the missing safety data; a smaller adapter is not a safety lever. section 5.1.4

For distillation between similarly sized teacher and student models, r=16 was the experimental sweet spot: r=8 had roughly three points lower retention in an earlier experiment, while r=32 gave no measurable improvement. A larger teacher–student behavioral gap may warrant testing r=32 or 64. section 7.7.2

Listing 5.2 Creating a LoRA configuration and passing it to SFTTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig
from trl import SFTTrainer
 
model_name = "Qwen/Qwen3-4B-Instruct-2507"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
 
# Define LoRA hyper-parameters
lora_config = LoraConfig(
    r=16,                # rank (try 8, 16, or 32)
    lora_alpha=32,       # scaling factor (typically 2x r)
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj",
                    "up_proj", "gate_proj", "down_proj"],  # attn + MLP
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)
 
# SFTTrainer applies LoRA automatically - no manual get_peft_model call
trainer = SFTTrainer(
    model=model,
    train_dataset=train_ds,
    eval_dataset=valid_ds,
    peft_config=lora_config,   # LoRA applied here
    ...
)
 
trainer.train()

This PEFT configuration sets r=16 and lora_alpha=32 before TRL’s SFTTrainer applies the adapter. section 5.1.2

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains rank’s capacity and cost tradeoffs, recommends a validation-driven selection process, and compares measured outcomes.
Examples
Compares IT-support adapters at ranks 8 and 16 and tests ranks 8, 16, and 32 in distillation.
What you can do after
Operate/configure LoRA rank by starting at 16, testing increases against validation performance, and checking safety separately.
How it connects
Connects rank to LoRA scaling, safety alignment, knowledge distillation, and supervised fine-tuning.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.