liveBook · Manning

Learning rate

As explained in LLM Customization and Fine-Tuning

The learning rate is a training hyperparameter that controls the size of parameter updates during optimization.

The learning rate controls the size of parameter updates during optimization. In full supervised fine-tuning (SFT), a rate that is too high can destabilize training or overwrite useful pretrained behavior; one that is too low can leave the model underfitting. section 6.1.2 section 6.2.2

For full SFT of 4B to 7B models, 2e-5 is a practical starting point, with 1e-5 to 5e-5 a typical range. Full SFT generally uses lower rates than LoRA because every parameter can change. A short warmup gradually increases the rate at the start, when gradients may be large or inconsistent. section 6.1.2 section 6.6.2

Table 6.1 Starting hyperparameters for SFT on 4B to 7B models
Parameter
Recommended
Notes
num_train_epochs
3
Rarely need more than 5; watch val loss
per_device_train_batch_size
1 to 2
Limited by GPU VRAM
gradient_accumulation_steps
8 to 16
An effective batch of 16 is a good default
learning_rate
2e-5
Range: 1e-5 to 5e-5
warmup_ratio
0.03 to 0.10
Or 50 to 200 fixed steps
weight_decay
0.01
Mild regularization
max_length
512 to 1024
Must cover your longest examples
bf16
True
Use fp16 only if bf16 unsupported
gradient_checkpointing
True
Essential for consumer GPUs
save_total_limit
2 to 3
Disk space: intermediate checkpoints are about 24 GB each with optimizer state; the final model is about 8 GB

Starting settings place the learning rate alongside batch size, warmup, and other constraints for full SFT. section 6.1.2

Tune the rate before other primary hyperparameters:

  • Diagnose the loss curves first. Flat, high training and validation losses can indicate a rate that is too low, but noisy data or truncated examples are other possible causes. section 6.9 section 6.2.2
  • Try 5e-6, 1e-5, 2e-5, and 5e-5 while holding epochs and batch size constant. Select the rate with the lowest validation loss, then tune epochs and effective batch size. section 6.9
  • If loss oscillates wildly or becomes NaN, reduce the rate by a factor of 5 to 10 and restart. If validation loss rises while training loss falls, stop at the epoch with the lowest validation loss. section 6.2.2
  • Stop tuning when the target is met or gains flatten. If primary parameter sweeps do not reach the target, invest in additional training data rather than continuing configuration changes. section 6.9

An illustrative tuning run

With epochs fixed at 3 and effective batch size at 16, lowering the rate from 2e-5 to 1e-5 reduced validation loss from 1.42 to 1.38 and raised accuracy from 67% to 69%. Raising it to 5e-5 instead increased validation loss to 1.51. Later changes brought accuracy to 74%, just short of a 75% target; diminishing gains pointed toward collecting more data. section 6.9

The appropriate rate also depends on the training stage. For direct preference optimization (DPO), 5e-6 can gently refine an already capable SFT model; a higher rate risks drifting from the reference policy and losing general capabilities. section 8.2.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Provides starting ranges, a one-variable sweep, loss-curve diagnostics, and stopping criteria for hyperparameter iteration.
Examples
Compares an illustrative six-run SFT log and a DPO run using 270 training preference pairs.
What you can do after
Operate/configure learning rates for fine-tuning and diagnose underfitting or instability using training and validation loss.
How it connects
Connects supervised fine-tuning, LoRA, catastrophic forgetting, validation loss, warmup, and direct preference optimization.

Also explained in

Compare how 23 books explain learning rate →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.