Supervised learning trains a model on paired inputs and target outputs. For conversational models, the pairing can be a question as input (X) and a manually written answer as output (y). Fine-tuning on these pairs changes a pre-trained GPT model from mainly continuing text into responding to prompts. section 5.2.1
Supervised fine-tuning for responses
After unsupervised pre-training on text, a GPT model is fine-tuned with questions and corresponding answers from manual labelers. The labeled examples teach the model to produce an answer for a question rather than simply predict a continuation. section 5.2.1
Figure 5.12 Demonstration of Supervised Fine-Tuning of a ChatGPT model.
The figure contrasts unlabeled text used for pre-training with X–y question-and-answer pairs used for supervised fine-tuning. section 5.2.1
Supervised fine-tuning is one stage in a broader training pipeline. It follows compute-intensive unsupervised pre-training and precedes reinforcement learning that uses labeler rankings or scores to train a reward model and further tune the model. section 5.2.1























