A hyperparameter is a configurable setting that controls training rather than a value learned from training data. In supervised fine-tuning (SFT), choices such as learning rate, batch size, and training epochs affect memory use, stability, and whether the model retains useful pretrained behavior. section 6.1.2 section 6.9
Full-parameter SFT needs tighter controls than LoRA: optimizer states and activations consume more memory, while aggressive updates can cause catastrophic forgetting. A micro-batch of 1 with 16 gradient accumulation steps gives an effective batch of 16 without holding all 16 examples’ activations at once. A learning rate of 2e-5 keeps updates conservative relative to a common LoRA setting of 2e-4. section 6.1.2
Table 6.1 Starting hyperparameters for SFT on 4B to 7B models
|
Parameter
|
Recommended
|
Notes
|
|
num_train_epochs
|
3
|
Rarely need more than 5; watch val loss
|
|
per_device_train_batch_size
|
1 to 2
|
Limited by GPU VRAM
|
|
gradient_accumulation_steps
|
8 to 16
|
An effective batch of 16 is a good default
|
|
learning_rate
|
2e-5
|
Range: 1e-5 to 5e-5
|
|
warmup_ratio
|
0.03 to 0.10
|
Or 50 to 200 fixed steps
|
|
weight_decay
|
0.01
|
Mild regularization
|
|
max_length
|
512 to 1024
|
Must cover your longest examples
|
|
bf16
|
True
|
Use fp16 only if bf16 unsupported
|
|
gradient_checkpointing
|
True
|
Essential for consumer GPUs
|
|
save_total_limit
|
2 to 3
|
Disk space: intermediate checkpoints
are about 24 GB each with optimizer state; the final model is about 8 GB
|
Use these starting settings for full SFT on 4B–7B models, adjusting for GPU memory and validation results. section 6.1.2
Tune from a diagnosed baseline, changing one variable per run:
- Check loss curves first: flat loss suggests underfitting, while rising validation loss alongside falling training loss suggests overfitting. section 6.9
- Try learning rates first, holding epochs and batch size constant; select the rate with the lowest validation loss. section 6.9
- Fix the learning rate and test epochs, stopping where validation loss plateaus. Adjust effective batch size next; use accumulation to increase it when loss is noisy. section 6.9
- Adjust warmup and weight decay only if earlier changes do not resolve the issue. Log each run and stop when the target is met or gains diminish. section 6.9
Recognizing a plateau
An illustrative six-run log starts at 67% accuracy against a 75% target and reaches 74%. The final gain is only 0.5 percentage points; rather than continuing to tune, the next step is to curate more training examples. section 6.9
Table 6.11 Example hyperparameter iteration log (illustrative)
|
Run
|
Change
|
LR
|
Epochs
|
Batch
|
Val Loss
|
Accuracy
|
Notes
|
|
1
|
Baseline
|
2e-5
|
3
|
16
|
1.42
|
67.0%
|
Below target
|
|
2
|
Lower LR
|
1e-5
|
3
|
16
|
1.38
|
69.0%
|
Slight improvement
|
|
3
|
Higher LR
|
5e-5
|
3
|
16
|
1.51
|
64.0%
|
Worse, too aggressive
|
|
4
|
Best LR + more epochs
|
1e-5
|
5
|
16
|
1.35
|
72.0%
|
Good improvement
|
|
5
|
Add weight decay
|
1e-5
|
5
|
16
|
1.33
|
73.5%
|
Marginal gain
|
|
6
|
Larger batch
|
1e-5
|
5
|
32
|
1.31
|
74.0%
|
Near target
|
Inspect the changes, validation losses, and accuracy across the illustrative runs. section 6.9
The available controls depend on the training platform. Azure OpenAI exposes epoch count, batch size, and a learning-rate multiplier while constraining choices such as optimizer and scheduler; its defaults can reduce configuration work. section 6.11.1

























