Low-rank adaptation (LoRA) specializes a model by training a small set of parameters while keeping its original weights frozen. Compared with full fine-tuning, this reduces training cost and time, but LoRA alone does not address the base model’s memory consumption. section 1.2.3 section 7.4
For a weight matrix W of size d × d, LoRA learns an update ΔW = AB, where A is d × r and B is r × d. The product has the same shape as W and can be added to it. Rank r controls the trade-off: a larger rank provides more expressive capacity but requires more trainable weights. Common starting ranks are 4, 8, and 16. section 7.1.1
Attention projection parameter count
In SmolLM2-1.7B, q_proj is 2048 × 2048. At rank 8, its LoRA matrices contain 2048 × 8 + 8 × 2048 = 32,768 parameters, versus 4,194,304 in the original matrix: a 99.2% reduction for that projection. section 7.1.1
Listing 7.2 LoRA applied to q_proj of SmolLM2-1.7B
import torch
d = 2048 #1
r = 8 #2
W = torch.randn(d, d) #3
A = torch.randn(d, r)
B = torch.zeros(r, d) #4
delta_W = A @ B
print(f"W dimensions: {W.shape}")
print(f"ΔW dimensions: {delta_W.shape}")
print()
params_W = d * d
params_lora = d * r + r * d
reduction = (1 - params_lora / params_W) * 100
print(f"Parameters in W: {params_W:,}")
print(f"Parameters in A + B: {params_lora:,}")
print(f"Reduction: {reduction:.1f}%")
B_trained = torch.randn(r, d) * 0.01 #5
delta_W_trained = A @ B_trained
W_adapted = W + delta_W_trained #6
print(f"\nW dimensions (unchanged): {W_adapted.shape}")
print(f"Sample weights before adaptation: {W[0, :4]}")
print(f"Sample weights after adaptation: {W_adapted[0, :4]}")
Check the matrix shapes and parameter reduction in PyTorch; the final update is simulated rather than trained. section 7.1.1
LoRA can follow structural optimization and knowledge recovery: prune the model, recover general capabilities through distillation, then fine-tune on domain data. For example, financial reports can guide pruning and later specialize a recovered model for earnings analysis. section 1.2.5
A function-calling workflow illustrates the implementation choices:
- Load a post-trained Qwen3-0.6B checkpoint in float16. Disable the cache for gradient checkpointing, match the padding token, and enable input gradients. section 13.3
- Configure PEFT with rank 16, alpha 32, dropout 0.05, and attention and MLP projection targets. This adapts around 1.5% of the model’s parameters. section 13.3
- Train for three epochs with a learning rate of 2e-4 and an effective batch size of 8. Compute loss only on completions so training focuses on generating the function call. section 13.3
- Save the adapter, merge it into the specialist model, restore the cache, disable gradient checkpointing, and switch to evaluation mode. section 13.3
Listing 13.6 LoRA configuration
lora_config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=[
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
],
)
train_model = get_peft_model(train_model, lora_config)
train_model.print_trainable_parameters()Inspect the PEFT configuration and the exact projection modules selected for adaptation. section 13.3
For tighter memory constraints, QLoRA combines adaptation with 4-bit NormalFloat quantization. DoRA instead separates weight magnitude and direction for training, addressing a difference between LoRA and full fine-tuning. section 7.4
















