A fine-tuning run can look successful because training loss falls while performance on new inputs deteriorates. Validation loss measures the training objective on held-out data, making it a primary signal for whether learned behavior generalizes beyond the training examples. Lower loss is better, but its trajectory matters more than a single reading. section 6.2 section 6.2.2
Plot training and validation loss together to guide the next decision:
- Both decline and level off near each other: learning is converging. A gap below roughly 0.1–0.2 is a healthy starting point. section 6.2.2
- Training loss falls while validation loss plateaus or rises: memorization may be overtaking transferable learning. Stop when validation loss worsens and use the checkpoint with the lowest validation loss. section 6.2 section 6.2.2
- Both remain high and flat: investigate underfitting, including a low learning rate, noisy data, or a maximum sequence length that truncates the assistant response. section 6.2.2
- Loss oscillates or becomes NaN: investigate instability; reducing the learning rate by a factor of 5–10 and restarting is a suggested response. section 6.2.2
Figure 6.3 The four training loss patterns encountered during fine-tuning.

Compare the four loss-curve patterns before deciding whether to continue, stop, or investigate the run. section 6.2.2
For 4B–7B models trained on 400–1,000 examples, a validation-minus-training loss gap of 0.15–0.30 is a warning to reduce epochs or add data; a gap above 0.30 suggests stopping for likely overfitting. These are starting points, not universal rules. Process the validation split consistently with the training split so the losses measure the same answer style. section 6.2.3 section 5.1.2
A LoRA run starts to memorize
In a 450-example, three-epoch run, training loss fell from 2.32 to 1.38. Validation loss stayed near 1.71 for two epochs, then rose to 1.77. The divergence suggests reducing epochs or adding data rather than assuming that lower training loss means a better adapter. section 5.1.2
To limit overfitting, configure early stopping and retain the best checkpoint. Other options include more diverse examples, weight decay or dropout, and switching from full fine-tuning to LoRA when model capacity is too high for the dataset. For hyperparameter iteration, change one setting per run: select a learning rate using validation loss, then test epochs and stop when validation loss plateaus. section 6.8.2 section 6.9
Listing 6.4 Training monitor (chapter06/scripts/monitor.py)
def monitor(run_dir: str) -> None:
state_file = Path(run_dir) / "trainer_state.json"
state = json.loads(state_file.read_text())
history = state.get("log_history", [])
# Separate training and eval entries
train_entries = [e for e in history if "loss" in e and "eval_loss" not in e]
eval_entries = [e for e in history if "eval_loss" in e]
# ... (print first/latest training loss and change) ...
# Warn if overfitting #A
if len(eval_entries) >= 2:
recent = eval_entries[-1]["eval_loss"]
previous = eval_entries[-2]["eval_loss"]
if recent > previous + 0.02:
print(" WARNING: Validation loss increased"
" between last two evals.")
print(" This may indicate overfitting. Consider stopping early.")
# Gradient norm check #B
grad_entries = [e for e in history if "grad_norm" in e]
if grad_entries:
max_norm = max(e["grad_norm"] for e in grad_entries)
if max_norm > 5.0:
print(" WARNING: Gradient norm exceeded 5.0"
" -- consider reducing learning rate.")Inspect a training monitor that reads eval_loss from trainer_state.json and warns when validation loss rises between evaluations. section 6.2.2
Validation loss is a training-health signal, not a complete deployment decision. Compare the fine-tuned model with the base model against predefined success thresholds, and run safety regression tests after training. section 6.15






