Reinforcement learning tunes a model toward outcomes that receive higher reward. In ChatGPT training, it follows unsupervised pretraining and supervised fine-tuning, using human judgments to improve generated answers. section 5.2.1
A reward-guided answer-generation loop works as follows:
- Labelers rank, score, or choose between candidate answers to questions. section 5.2.1
- A reward model learns to score a question-and-answer pair from those judgments. section 5.2.1
- GPT generates an answer word by word; its answer and the question are scored by the reward model. section 5.2.1
- The score produces a loss that is backpropagated to tune GPT parameters toward higher-reward answer sequences. section 5.2.1
Figure 5.13 Demonstration of Reinforcement Learning using Human Feedback (RLHF) to train ChatGPT
This workflow connects human answer preferences, reward-model scoring, generated sequences, and parameter updates. section 5.2.1



























