Fine-tuning adapts a pretrained model by training it further on more specific data or objectives. It turns broad language ability into behavior suited to particular tasks, instructions, domains, or conversational goals. section 1.6
A typical fine-tuning workflow has three stages:
- Start with a pretrained neural network, such as GPT-3, whose general language capability comes from large-scale pretraining. section 1.6
- Provide many training examples containing prompts and desired completions, using supervised learning to demonstrate the target behavior. section 1.6
- Adjust the model’s parameters through repeated backpropagation-like learning passes until its outputs move toward the desired completions. section 1.6
From sentence completion to dialogue
A model initially trained to predict likely next words can be fine-tuned on human conversation examples. It then produces more coherent, relevant question-and-answer dialogue rather than generic sentence completions. ChatGPT’s conversational capabilities are described as receiving this supervised refinement, followed by reinforcement learning from human feedback. section 1.5
Figure 1.13 Enhancing a pretrained model through fine-tuning
Use the diagram to visualize how additional training enhances a pretrained model. section 1.6
Fine-tuning can specialize a model for an underrepresented domain, such as a branch of medicine, or improve adherence to particular instructions. It does not by itself make pretrained knowledge current; applications may still need updated-information retrieval and other supporting components. section 1.6 section 1.7
