LoRA rank (r) sets the dimensionality of an adapter’s low-rank updates and controls its expressivity. Higher rank can represent more complex changes, but typically costs more GPU memory and training time. Rank also affects the update’s scaling: effective weight = W + (α/r) × A × B, where α is the scaling factor. section 5.3
Choose rank against the task and validation results:
- Try r=4 or 8 for relatively simple changes such as terminology, shorter answers, or a fixed format. section 5.3
- Start at r=16 for a general-purpose baseline. Try r=32 if validation performance plateaus; if it does not improve results, rank may not be the bottleneck. section 5.3
- Consider a rank increase before full supervised fine-tuning when LoRA is already within roughly five percentage points of the target metric. section 6.6.1
Table 5.4 Impact of LoRA rank on performance and resources on a sample task
|
Rank (r)
|
Alpha (α)
|
Training Time
|
Val Accuracy
|
VRAM
|
Recommendation
|
|
4
|
8
|
1.2 hr
|
61.3%
|
9.2 GB
|
Too limited for complex
tasks
|
|
8
|
16
|
1.4 hr
|
64.8%
|
9.8 GB
|
Good for simple
style/format tasks
|
|
16
|
32
|
1.8 hr
|
67.8%
|
10.4 GB
|
Sweet spot for most use
cases
|
|
32
|
64
|
2.4 hr
|
68.1%
|
11.9 GB
|
Marginal gain (+0.3pp) for
33% more time
|
|
64
|
128
|
3.8 hr
|
68.3%
|
14.2 GB
|
Diminishing returns,
approaching full SFT cost
|
|
Full SFT
|
N/A
|
18+ hr
|
69.1%
|
32+ GB
|
Only +1.3pp over r=16 at
~10x cost; not worth it
|
On a sample 4B-model task, higher ranks add cost for diminishing validation gains. These figures are a comparison for that task, not guarantees for another workload. section 5.3
A smaller rank did not restore safety
An IT-support adapter trained on 450 helpful examples with no refusal examples reduced the safety refusal rate from 100% to 60%. Retraining at r=8 instead of r=16 left refusal at 60%. Token-F1 was 0.167 versus 0.154, but the difference was within noise on the 50-example test split. Reducing rank did not fix the missing safety data; a smaller adapter is not a safety lever. section 5.1.4
For distillation between similarly sized teacher and student models, r=16 was the experimental sweet spot: r=8 had roughly three points lower retention in an earlier experiment, while r=32 gave no measurable improvement. A larger teacher–student behavioral gap may warrant testing r=32 or 64. section 7.7.2
Listing 5.2 Creating a LoRA configuration and passing it to SFTTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig
from trl import SFTTrainer
model_name = "Qwen/Qwen3-4B-Instruct-2507"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")
# Define LoRA hyper-parameters
lora_config = LoraConfig(
r=16, # rank (try 8, 16, or 32)
lora_alpha=32, # scaling factor (typically 2x r)
target_modules=["q_proj", "v_proj", "k_proj", "o_proj",
"up_proj", "gate_proj", "down_proj"], # attn + MLP
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)
# SFTTrainer applies LoRA automatically - no manual get_peft_model call
trainer = SFTTrainer(
model=model,
train_dataset=train_ds,
eval_dataset=valid_ds,
peft_config=lora_config, # LoRA applied here
...
)
trainer.train()This PEFT configuration sets r=16 and lora_alpha=32 before TRL’s SFTTrainer applies the adapter. section 5.1.2
