Catastrophic forgetting occurs when fine-tuning a pretrained model on new data or tasks causes it to lose previously learned knowledge, capabilities, or reasoning skills. The risk matters when adapting one general-purpose model for specialized work without sacrificing earlier performance. section 2.8.8 section 9.6
Model adaptation creates the risk and suggests the mitigation:
- Fine-tuning on a new dataset can overwrite original parameters with task-specific ones, reducing general knowledge and performance on previous tasks. section 9.6
- Parameter-efficient fine-tuning updates only a small subset of parameters while keeping most pretrained parameters fixed, helping preserve generalization. section 9.6
Specializing a language model
Fine-tuning an LLM for sentiment analysis, spam detection, question answering, summarization, or translation may improve the target task while causing the model to forget knowledge and reasoning learned during pretraining. section 2.8.8
Table 9.2 Model adaptation techniques
|
Technique
|
Description
|
|---|---|
|
Prompt tuning
|
Prompt tuning is a technique for adapting LLMS to different tasks by providing specific cues or prompts that guide their generation or prediction. It does not require retraining the model or updating its weights, which makes it faster and cheaper than fine-tuning. It is particularly useful for tasks where only a small amount of adaptation is required.
|
|
Adapter modules
|
Adapter models are used in LLM fine-tuning to add small and task-specific modules (small neural networks) to the pretrained model and train only these modules on the task-specific data. They are also flexible and modular, as they can be easily added or removed for different tasks without affecting the pretrained model.
|
|
Bias-only (BitFit)
|
Bias-only (BitFit) is a technique for fine-tuning LLMs by modifying only the bias terms of the model or a subset of them. It offers a minimalistic approach to adaptation, requiring even fewer trainable parameters than LoRA. BitFit is based on fine-tuning, mainly exposing the knowledge learned by pretraining rather than acquiring new task-specific knowledge.
|
|
Layer freezing
|
Layer freezing is a fine-tuning technique that keeps some of the model layers fixed and only updates the rest. This method allows for more control over which aspects of the model are adapted and can reduce training time.
|
|
Knowledge distillation
|
Involves training a smaller, more efficient model (student) to mimic the behavior of a larger pretrained model (teacher). This method is useful for deploying LLMs in resource-constrained environments.
|
|
Meta-learning
|
Focuses on training the model to learn new tasks quickly with minimal data; often involves training on various tasks so the model can more efficiently adapt to new, unseen tasks
|
|
Differential privacy fine-tuning
|
Incorporates privacy-preserving techniques during fine-tuning to protect sensitive data. This is essential for applications where data privacy and security are paramount.
|
|
Reinforcement learning from human feedback (RLHF)
|
Involves fine-tuning models based on feedback or rewards derived from human interactions or evaluations. It is useful for tasks where human judgment is crucial, such as content moderation.
|
Compare adaptation approaches such as prompt tuning, adapter modules, and bias-only fine-tuning alongside PEFT-related trade-offs. section 9.6
