A general-purpose language model may struggle with an enterprise’s specialized vocabulary or tasks. Model adaptation starts with a pretrained model and adjusts it to perform better for a particular task, dataset, or domain, rather than training a new model from scratch. section 9.1
Choose the least demanding approach that meets the use case:
- Start with an existing SaaS capability where suitable. Otherwise, try prompt engineering and, when grounding is needed, retrieval-augmented generation (RAG). These approaches can complement later fine-tuning. section 9.0 section 9.8
- For fine-tuning, full fine-tuning updates all model parameters. Parameter-efficient fine-tuning (PEFT) updates fewer; low-rank adaptation (LoRA) trains added low-rank components instead of updating every weight. section 9.1.1 section 9.6
- Plan for clean, representative task data, computational cost, evaluation, and repeated iterations before production deployment. Fine-tuning can also impair performance on earlier tasks. section 9.1.2 section 9.6 section 9.8
Specializing a general model
A model trained on general text may perform poorly on medical questions. Fine-tuning it on a smaller, relevant dataset can help it learn patterns needed for that domain. Similarly, moving from sentiment analysis to emotion detection is task adaptation; moving from general to medical news is domain adaptation. section 9.1 section 9.1
Table 9.2 Model adaptation techniques
|
Technique
|
Description
|
|---|---|
|
Prompt tuning
|
Prompt tuning is a technique for adapting LLMS to different tasks by providing specific cues or prompts that guide their generation or prediction. It does not require retraining the model or updating its weights, which makes it faster and cheaper than fine-tuning. It is particularly useful for tasks where only a small amount of adaptation is required.
|
|
Adapter modules
|
Adapter models are used in LLM fine-tuning to add small and task-specific modules (small neural networks) to the pretrained model and train only these modules on the task-specific data. They are also flexible and modular, as they can be easily added or removed for different tasks without affecting the pretrained model.
|
|
Bias-only (BitFit)
|
Bias-only (BitFit) is a technique for fine-tuning LLMs by modifying only the bias terms of the model or a subset of them. It offers a minimalistic approach to adaptation, requiring even fewer trainable parameters than LoRA. BitFit is based on fine-tuning, mainly exposing the knowledge learned by pretraining rather than acquiring new task-specific knowledge.
|
|
Layer freezing
|
Layer freezing is a fine-tuning technique that keeps some of the model layers fixed and only updates the rest. This method allows for more control over which aspects of the model are adapted and can reduce training time.
|
|
Knowledge distillation
|
Involves training a smaller, more efficient model (student) to mimic the behavior of a larger pretrained model (teacher). This method is useful for deploying LLMs in resource-constrained environments.
|
|
Meta-learning
|
Focuses on training the model to learn new tasks quickly with minimal data; often involves training on various tasks so the model can more efficiently adapt to new, unseen tasks
|
|
Differential privacy fine-tuning
|
Incorporates privacy-preserving techniques during fine-tuning to protect sensitive data. This is essential for applications where data privacy and security are paramount.
|
|
Reinforcement learning from human feedback (RLHF)
|
Involves fine-tuning models based on feedback or rewards derived from human interactions or evaluations. It is useful for tasks where human judgment is crucial, such as content moderation.
|
Compare prompt tuning, adapter modules, and bias-only fine-tuning by what each changes in the model. section 9.6
Anchor adaptation in a specific business use case rather than treating fine-tuning as the default. Its potential gains in relevance and efficiency must be weighed against data needs, costs, evaluation results, and ethical considerations. section 9.1.2 section 9.8
