liveBook · Manning

Catastrophic forgetting

As explained in LLM Customization and Fine-Tuning

The loss of previously learned capabilities when a model is fine-tuned on narrower data, as updates overwrite parameters supporting older skills.

A fine-tuned model can improve on a domain task while losing general-purpose skills or safety behaviors it previously handled correctly. This loss, called catastrophic forgetting, occurs when updates from narrower training data overwrite parameters supporting older capabilities. section 6.6.2 section 6.8.1

Full supervised fine-tuning (SFT) is more exposed than LoRA because all parameters can change, rather than a constrained set of low-rank updates. Choose full SFT with that risk and its higher operational costs in mind; broader capability and safety checks remain necessary even when domain accuracy improves. section 6.6.2 section 6.8.1

Reduce and detect forgetting during an SFT run:

  • Use a conservative learning rate, such as 2e-5 or below, and stop early based on validation loss. Limit training to 3–5 epochs; validation loss typically plateaus by epoch 3 on datasets of 400–1,000 examples. section 6.6.2 section 6.8.1
  • Mix 10–20% general-purpose examples into the domain data as rehearsal. Compare a general-purpose golden set before and after training, and run a safety regression suite after every SFT run. section 6.8.1
  • For incremental retraining, retain a replay slice of original data and avoid letting one new domain dominate. Compare the retrained model with both the current production model and the base model before promotion; retain a rollback path. section 9.7.2

Alternative for sequential training

Self-Distillation Fine-Tuning (SDFT) uses the model’s in-context learning as a training signal. Reported benchmark results put prior-capability losses at roughly 1 point for SDFT versus 12 points for standard SFT. The tradeoff is about 2.5 times the compute and a need for strong in-context learning capability in a model of 7B or more parameters; the technique is too new for prescriptive recommendations. section 6.8.1 section 9.7.2

Figure 9.5 The incremental retraining loop: merge, resume from the production checkpoint, evaluate, and gate before deploying.

Follow the incremental retraining loop from data merging through evaluation and deployment gating. section 9.7.2

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains the overwrite mechanism and gives concrete prevention, evaluation, and deployment steps.
Examples
Uses general-purpose golden sets and reported self-distillation benchmark comparisons to illustrate evaluation and alternatives.
What you can do after
Diagnose capability loss and safety regression by comparing general-purpose and safety tests before and after fine-tuning.
How it connects
Connects supervised fine-tuning, LoRA, rehearsal, safety regression, and self-distillation fine-tuning.

Also explained in

Compare how 5 books explain catastrophic forgetting →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.