liveBook · Manning

Validation loss

As explained in LLM Customization and Fine-Tuning

A measure of a model’s training objective on held-out data, used to assess how well it generalizes beyond its training examples.

A fine-tuning run can look successful because training loss falls while performance on new inputs deteriorates. Validation loss measures the training objective on held-out data, making it a primary signal for whether learned behavior generalizes beyond the training examples. Lower loss is better, but its trajectory matters more than a single reading. section 6.2 section 6.2.2

Plot training and validation loss together to guide the next decision:

  • Both decline and level off near each other: learning is converging. A gap below roughly 0.1–0.2 is a healthy starting point. section 6.2.2
  • Training loss falls while validation loss plateaus or rises: memorization may be overtaking transferable learning. Stop when validation loss worsens and use the checkpoint with the lowest validation loss. section 6.2 section 6.2.2
  • Both remain high and flat: investigate underfitting, including a low learning rate, noisy data, or a maximum sequence length that truncates the assistant response. section 6.2.2
  • Loss oscillates or becomes NaN: investigate instability; reducing the learning rate by a factor of 5–10 and restarting is a suggested response. section 6.2.2
Figure 6.3 The four training loss patterns encountered during fine-tuning.

Compare the four loss-curve patterns before deciding whether to continue, stop, or investigate the run. section 6.2.2

For 4B–7B models trained on 400–1,000 examples, a validation-minus-training loss gap of 0.15–0.30 is a warning to reduce epochs or add data; a gap above 0.30 suggests stopping for likely overfitting. These are starting points, not universal rules. Process the validation split consistently with the training split so the losses measure the same answer style. section 6.2.3 section 5.1.2

A LoRA run starts to memorize

In a 450-example, three-epoch run, training loss fell from 2.32 to 1.38. Validation loss stayed near 1.71 for two epochs, then rose to 1.77. The divergence suggests reducing epochs or adding data rather than assuming that lower training loss means a better adapter. section 5.1.2

To limit overfitting, configure early stopping and retain the best checkpoint. Other options include more diverse examples, weight decay or dropout, and switching from full fine-tuning to LoRA when model capacity is too high for the dataset. For hyperparameter iteration, change one setting per run: select a learning rate using validation loss, then test epochs and stop when validation loss plateaus. section 6.8.2 section 6.9

Listing 6.4 Training monitor (chapter06/scripts/monitor.py)
def monitor(run_dir: str) -> None:
    state_file = Path(run_dir) / "trainer_state.json"
    state = json.loads(state_file.read_text())
    history = state.get("log_history", [])
 
    # Separate training and eval entries
    train_entries = [e for e in history if "loss" in e and "eval_loss" not in e]
    eval_entries = [e for e in history if "eval_loss" in e]
 
    # ... (print first/latest training loss and change) ...
 
    # Warn if overfitting                                                #A
    if len(eval_entries) >= 2:
        recent = eval_entries[-1]["eval_loss"]
        previous = eval_entries[-2]["eval_loss"]
        if recent > previous + 0.02:
            print("  WARNING: Validation loss increased"
                  " between last two evals.")
            print("  This may indicate overfitting. Consider stopping early.")
 
    # Gradient norm check                                                #B
    grad_entries = [e for e in history if "grad_norm" in e]
    if grad_entries:
        max_norm = max(e["grad_norm"] for e in grad_entries)
        if max_norm > 5.0:
            print("  WARNING: Gradient norm exceeded 5.0"
                  " -- consider reducing learning rate.")

Inspect a training monitor that reads eval_loss from trainer_state.json and warns when validation loss rises between evaluations. section 6.2.2

Validation loss is a training-health signal, not a complete deployment decision. Compare the fine-tuned model with the base model against predefined success thresholds, and run safety regression tests after training. section 6.15

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains loss-curve patterns, conditional gap thresholds, and remedies, with a monitor that reads logged evaluation loss.
Examples
Illustrates loss divergence in a 450-example LoRA run and checkpoint selection in a three-epoch full fine-tuning run.
What you can do after
Diagnose training runs with loss curves and select a checkpoint or corrective action when validation loss stops improving.
How it connects
Connects validation loss to training loss, overfitting, underfitting, early stopping, and LoRA.

Also explained in

Compare how 7 books explain validation loss →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.