Catastrophic forgetting is the loss of broad pretrained capability after adapting a model to a narrow domain or task. It is a central trade-off in fine-tuning: stronger specialization can destabilize representations that previously supported varied general-purpose tasks. section 3.3.1 section 3.8
Full parameter fine-tuning is most exposed because gradient descent reshapes every layer. A model trained intensively on legal English, for example, may become a stronger legal specialist while faltering outside that domain—like a polyglot losing fluency in other languages after exclusive retraining. This approach suits critical applications with large representative datasets and ample compute, but deliberately accepts reduced versatility. section 3.3.1
Figure 3.7 Full parameter fine-tuning. The base model is adapted using a domain-specific training dataset, with gradient descent propagating through all layers. The resulting fine-tuned model retains the original architecture but has every weight updated — unlike LoRA or other PEFT methods, nothing is frozen.

Use the full-parameter training flow to see why forgetting is a risk: every weight is updated and none are frozen. section 3.3.1
Ways to reduce the risk include:
- Freeze a substantial portion of the network with half fine-tuning (HFT), preserving broad pretrained knowledge while training enough layers for task-specific adaptation. Select frozen layers through experimentation and validation, commonly beginning with lower layers frozen and unfreezing higher layers gradually. section 3.4.1
- During unsupervised domain adaptation, mix domain data with a sample of the original pretraining distribution—typically 5–20%—so narrow domain exposure does not displace general versatility. section 3.6
Domain adaptation without over-specializing
A masked-language-model adaptation example uses clinical text about acute myocardial infarction, troponin levels, electrocardiograms, aspirin, and nitroglycerin. It illustrates domain immersion; retaining some original-distribution data is the suggested counterweight to catastrophic forgetting. section 3.6




