15 Regularization
This chapter covers
- Constraining the RLHF process with KL divergence
- Using regularization to prevent models from producing nonsensical outputs
- Why RL generalizes better than SFT through implicit regularization
- Other regularization techniques for training LLMs
In this book, we’ve covered many tools for modifying the model to learn from human preferences, verifiable rewards, and other valuable signals. All the methods we use are very powerful and can cause the model to change too much relative to the strong, general model from the previous training stage (often called the reference model). When the model learns too much from a given reward, causing out-of-distribution performance to drop, this is called over-optimization (as we discussed in the previous chapter).
Throughout the RLHF optimization, many regularization steps are used to prevent over-optimization of the reward model. Over-optimization in these contexts looks like models that output nonsensical text. Some examples of optimization “off the rails” are models that output followable math reasoning with extremely incorrect answers, repeated text, switching languages, or excessive special characters. This chapter covers the different methods used to control the optimization of models.