liveBook · Manning

LoRA scaling factor

As explained in LLM Customization and Fine-Tuning

A LoRA hyperparameter that scales the contribution of the low-rank adapter to the model.

The LoRA scaling factor, α, controls how strongly a low-rank adapter contributes to a model’s effective weights. Its effect depends on the ratio α/r, where r is the adapter rank; changing α without considering rank changes that ratio. section 5.3

effective_weight = W + (α/r) × A×B

W is the base weight, A×B is the low-rank adapter contribution, and α/r scales that contribution. section 5.3

A practical starting configuration is r=16 and α=32, giving α/r=2. The configuration example describes α as typically twice the rank. Rank is a separate decision: it controls adapter expressivity, with higher ranks increasing memory use and training time. section 5.3 section 5.1.2

Configure an IT support adapter

For an IT support fine-tuning run, use --lora_r 16 and --lora_alpha 32, the training script’s defaults. In PEFT, the corresponding LoraConfig fields are r=16 and lora_alpha=32; TRL’s SFTTrainer applies the adapter while training on the prepared data. section 5.1.2

If validation performance plateaus, try increasing rank from 16 to 32. The sample configurations pair r=32 with α=64, preserving the α/r ratio of 2. Their results change rank and α together, so they do not isolate the effect of α alone. section 5.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains the α/r weight multiplier and a typical setting, but does not isolate α’s effect in the reported comparisons.
Examples
Uses an IT support adapter with r=16 and α=32, alongside sample rank and resource comparisons.
What you can do after
Apply it via PEFT’s LoraConfig and TRL’s SFTTrainer when training a LoRA adapter.
How it connects
Connects LoRA scaling to adapter rank, effective weights, and validation performance.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.