Rank, written r, sets the inner dimension of LoRA’s trainable matrices A and B. Their product has the same dimensions as the frozen weight matrix, so it can be added to that matrix. section 7.1.1
Choose rank with the capacity–cost trade-off in mind:
- Increasing rank gives the update more expressive capacity but increases the number of trainable weights. section 7.1.1
- Values 4, 8, and 16 are common starting points; rank 8 is a reasonable default until evidence supports a higher value. section 7.1.1
- The appropriate value remains an empirical decision; DoRA is described as able to maintain performance at lower ranks than LoRA. section 7.1.1 section 7.1.2
Rank 8 for a 2048 × 2048 projection
For q_proj, rank 8 gives A and B a combined 32,768 parameters, compared with 4,194,304 in the original matrix—a 99.2% reduction. section 7.1.1
Figure 7.1 LoRA decomposes the weight matrix W (2048×2048) into two low-rank matrices A and B. Together they contain 32,768 parameters — a 99.2% reduction compared to the 4,194,304 of the original matrix.
The figure shows how two low-rank matrices form an update with the original matrix’s dimensions while using fewer parameters. section 7.1.1
