The learning rate controls the size of parameter updates during optimization. In full supervised fine-tuning (SFT), a rate that is too high can destabilize training or overwrite useful pretrained behavior; one that is too low can leave the model underfitting. section 6.1.2 section 6.2.2
For full SFT of 4B to 7B models, 2e-5 is a practical starting point, with 1e-5 to 5e-5 a typical range. Full SFT generally uses lower rates than LoRA because every parameter can change. A short warmup gradually increases the rate at the start, when gradients may be large or inconsistent. section 6.1.2 section 6.6.2
Table 6.1 Starting hyperparameters for SFT on 4B to 7B models
|
Parameter
|
Recommended
|
Notes
|
|
num_train_epochs
|
3
|
Rarely need more than 5; watch val loss
|
|
per_device_train_batch_size
|
1 to 2
|
Limited by GPU VRAM
|
|
gradient_accumulation_steps
|
8 to 16
|
An effective batch of 16 is a good default
|
|
learning_rate
|
2e-5
|
Range: 1e-5 to 5e-5
|
|
warmup_ratio
|
0.03 to 0.10
|
Or 50 to 200 fixed steps
|
|
weight_decay
|
0.01
|
Mild regularization
|
|
max_length
|
512 to 1024
|
Must cover your longest examples
|
|
bf16
|
True
|
Use fp16 only if bf16 unsupported
|
|
gradient_checkpointing
|
True
|
Essential for consumer GPUs
|
|
save_total_limit
|
2 to 3
|
Disk space: intermediate checkpoints
are about 24 GB each with optimizer state; the final model is about 8 GB
|
Starting settings place the learning rate alongside batch size, warmup, and other constraints for full SFT. section 6.1.2
Tune the rate before other primary hyperparameters:
- Diagnose the loss curves first. Flat, high training and validation losses can indicate a rate that is too low, but noisy data or truncated examples are other possible causes. section 6.9 section 6.2.2
- Try 5e-6, 1e-5, 2e-5, and 5e-5 while holding epochs and batch size constant. Select the rate with the lowest validation loss, then tune epochs and effective batch size. section 6.9
- If loss oscillates wildly or becomes NaN, reduce the rate by a factor of 5 to 10 and restart. If validation loss rises while training loss falls, stop at the epoch with the lowest validation loss. section 6.2.2
- Stop tuning when the target is met or gains flatten. If primary parameter sweeps do not reach the target, invest in additional training data rather than continuing configuration changes. section 6.9
An illustrative tuning run
With epochs fixed at 3 and effective batch size at 16, lowering the rate from 2e-5 to 1e-5 reduced validation loss from 1.42 to 1.38 and raised accuracy from 67% to 69%. Raising it to 5e-5 instead increased validation loss to 1.51. Later changes brought accuracy to 74%, just short of a 75% target; diminishing gains pointed toward collecting more data. section 6.9
The appropriate rate also depends on the training stage. For direct preference optimization (DPO), 5e-6 can gently refine an already capable SFT model; a higher rate risks drifting from the reference policy and losing general capabilities. section 8.2.3






















