liveBook · Manning

Hyperparameter

As explained in LLM Customization and Fine-Tuning

A configurable setting that controls model training rather than being learned directly from training data.

A hyperparameter is a configurable setting that controls training rather than a value learned from training data. In supervised fine-tuning (SFT), choices such as learning rate, batch size, and training epochs affect memory use, stability, and whether the model retains useful pretrained behavior. section 6.1.2 section 6.9

Full-parameter SFT needs tighter controls than LoRA: optimizer states and activations consume more memory, while aggressive updates can cause catastrophic forgetting. A micro-batch of 1 with 16 gradient accumulation steps gives an effective batch of 16 without holding all 16 examples’ activations at once. A learning rate of 2e-5 keeps updates conservative relative to a common LoRA setting of 2e-4. section 6.1.2

Table 6.1 Starting hyperparameters for SFT on 4B to 7B models
Parameter
Recommended
Notes
num_train_epochs
3
Rarely need more than 5; watch val loss
per_device_train_batch_size
1 to 2
Limited by GPU VRAM
gradient_accumulation_steps
8 to 16
An effective batch of 16 is a good default
learning_rate
2e-5
Range: 1e-5 to 5e-5
warmup_ratio
0.03 to 0.10
Or 50 to 200 fixed steps
weight_decay
0.01
Mild regularization
max_length
512 to 1024
Must cover your longest examples
bf16
True
Use fp16 only if bf16 unsupported
gradient_checkpointing
True
Essential for consumer GPUs
save_total_limit
2 to 3
Disk space: intermediate checkpoints are about 24 GB each with optimizer state; the final model is about 8 GB

Use these starting settings for full SFT on 4B–7B models, adjusting for GPU memory and validation results. section 6.1.2

Tune from a diagnosed baseline, changing one variable per run:

  • Check loss curves first: flat loss suggests underfitting, while rising validation loss alongside falling training loss suggests overfitting. section 6.9
  • Try learning rates first, holding epochs and batch size constant; select the rate with the lowest validation loss. section 6.9
  • Fix the learning rate and test epochs, stopping where validation loss plateaus. Adjust effective batch size next; use accumulation to increase it when loss is noisy. section 6.9
  • Adjust warmup and weight decay only if earlier changes do not resolve the issue. Log each run and stop when the target is met or gains diminish. section 6.9

Recognizing a plateau

An illustrative six-run log starts at 67% accuracy against a 75% target and reaches 74%. The final gain is only 0.5 percentage points; rather than continuing to tune, the next step is to curate more training examples. section 6.9

Table 6.11 Example hyperparameter iteration log (illustrative)
Run
Change
LR
Epochs
Batch
Val Loss
Accuracy
Notes
1
Baseline
2e-5
3
16
1.42
67.0%
Below target
2
Lower LR
1e-5
3
16
1.38
69.0%
Slight improvement
3
Higher LR
5e-5
3
16
1.51
64.0%
Worse, too aggressive
4
Best LR + more epochs
1e-5
5
16
1.35
72.0%
Good improvement
5
Add weight decay
1e-5
5
16
1.33
73.5%
Marginal gain
6
Larger batch
1e-5
5
32
1.31
74.0%
Near target

Inspect the changes, validation losses, and accuracy across the illustrative runs. section 6.9

The available controls depend on the training platform. Azure OpenAI exposes epoch count, batch size, and a learning-rate multiplier while constraining choices such as optimizer and scheduler; its defaults can reduce configuration work. section 6.11.1

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains individual settings and their tradeoffs, then gives an ordered, one-variable-at-a-time tuning procedure with stopping criteria.
Examples
Uses starting settings for 4B–7B models and an illustrative six-run accuracy log that plateaus below target.
What you can do after
Diagnose and tune full-parameter SFT runs using loss curves and one-variable comparisons, then consider more training data when gains plateau.
How it connects
Connects hyperparameters to gradient accumulation, validation loss, overfitting, catastrophic forgetting, and LoRA.

Also explained in

Compare how 26 books explain hyperparameter →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.