Low-rank approximation represents a large matrix with two smaller factors. For a pretrained weight matrix W₀, LoRA replaces a full update ΔW with ΔW = BA, where B ∈ ℝᵈˣʳ, A ∈ ℝʳˣᵏ, and r < min(d, k). The pretrained weights remain frozen while training learns the compact factors. section 8.3.1
The approximation creates practical trade-offs:
- Lower training cost: learning B and A requires fewer trainable parameters than directly modifying W₀. section 8.3.1
- Rank controls capacity: smaller r produces a more compact update, while larger r can represent more directions. section 8.3.1
- SVD can construct an illustrative rank-r approximation by retaining the largest singular values and corresponding singular vectors; training normally learns B and A instead. section 8.3.1
- LQ-LoRA separates a pretrained matrix into a high-precision low-rank component and a quantized component, updating only the former during fine-tuning. section 8.3.6
Listing 8.11 Illustrative example of LoRA
def lora_decomposition(W, r): B = np.zeros((W.shape[0], r)) #1 A = np.random.normal(0, 1, (r, W.shape[1])) U, S, Vt = np.linalg.svd(W, full_matrices=False) #2 B[:, :r] = U[:, :r] * np.sqrt(S[:r]) A[:r, :] = np.sqrt(S[:r])[:, np.newaxis] * Vt[:r, :] W_approx = B @ A #3 return B, A, W_approx d = 6 #4 W = np.random.randint(0, 10, size=(d, d)) #5 r = 5 # Rank for the decomposition #6 B, A, W_approx = lora_decomposition(W, r) #7
Run the matrix example to inspect how SVD-derived factors B and A reconstruct a rank-r approximation of W. section 8.3.1
Applying low-rank adaptation
A practical LoRA configuration sets lora_rank = 32, loads a Qwen3-4B base model with Unsloth, and attaches adapters to attention and feed-forward projections such as q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj. section 8.3.1
