6 Reinforcement learning
This chapter covers
- Using reinforcement learning to optimize human preferences
- Mathematics and intuitions for policy-
gradient algorithms - The derivations and tradeoffs of different
algorithms - New algorithms for reasoning models
- Implementation details of modern RLHF stacks
In the RLHF process, the reinforcement learning algorithm slowly updates the model’s weights with respect to feedback from a reward model. The policy—the model being trained—generates completions to prompts in the training set, then the reward model scores them, and the reinforcement learning optimizer takes gradient steps based on this information (see figure 6.1 for an overview). This chapter explains the mathematics and tradeoffs across various algorithms used to learn from the signal the reward model gives to on-policy data (i.e., data generated by the current version of the model being trained). These algorithms are run for a period of many epochs, often thousands or millions of batches across a larger set of prompts, with gradient updates in between each of them.