liveBook · Manning

Rank

As explained in Rearchitecting LLMs

Rank is a LoRA configuration hyperparameter that determines the dimensionality and expressive capacity of a low-rank update.

Rank, written r, sets the inner dimension of LoRA’s trainable matrices A and B. Their product has the same dimensions as the frozen weight matrix, so it can be added to that matrix. section 7.1.1

Choose rank with the capacity–cost trade-off in mind:

  • Increasing rank gives the update more expressive capacity but increases the number of trainable weights. section 7.1.1
  • Values 4, 8, and 16 are common starting points; rank 8 is a reasonable default until evidence supports a higher value. section 7.1.1
  • The appropriate value remains an empirical decision; DoRA is described as able to maintain performance at lower ranks than LoRA. section 7.1.1 section 7.1.2

Rank 8 for a 2048 × 2048 projection

For q_proj, rank 8 gives A and B a combined 32,768 parameters, compared with 4,194,304 in the original matrix—a 99.2% reduction. section 7.1.1

Figure 7.1 LoRA decomposes the weight matrix W (2048×2048) into two low-rank matrices A and B. Together they contain 32,768 parameters — a 99.2% reduction compared to the 4,194,304 of the original matrix.

The figure shows how two low-rank matrices form an update with the original matrix’s dimensions while using fewer parameters. section 7.1.1

How Rearchitecting LLMs covers it

How deep it goes
Explains the matrix factorization, parameter counts, and capacity trade-off, while leaving rank selection empirical.
Examples
Compares rank-8 parameters with the original q_proj matrix in SmolLM2-1.7B.
What you can do after
Evaluate rank choices by balancing update capacity against trainable parameter count.
How it connects
Connects LoRA rank to low-rank adaptation and DoRA’s ability to use smaller ranks.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.