Chapter 15: Regularization
Introduces regularization as a way to limit harmful changes during RLHF, especially by comparing the current policy with a reference policy.
After this chapter you will be able to
- Regularization and over-optimization. Explain why RLHF uses regularization: powerful optimization can change a model too much relative to the strong model from the previous training stage, causing performance outside the optimized setting to drop or outputs to become nonsensical.
- KL comparison with reference policy. Describe the chapter's most common regularization variant as a KL comparison between the current policy and a reference policy across generated samples, and distinguish the informal term KL distance from a true mathematical distance metric.
- KL divergence and model drift. Explain what KL divergence measures and identify the two model distributions compared in RLHF.
- Reverse KL reference penalty. Describe how the book's common reverse-KL penalty is estimated and what kind of departure from the reference it strongly penalizes.
- Static reference for KL penalties. Explain that the KL penalty compares the model’s generated tokens during training with a static reference model to encourage the trained model to retain a desired style. Identify the usual reference and one alternative.
- Sampled KL approximation. Explain why full sequence-level KL is impractical for LLMs and how sampling from the current model replaces a sum over possible completions with an expectation.
- Generated sequence log-probability comparison. Trace how the implementation uses a generated sequence to obtain policy and reference log-probabilities and approximate KL.
- Implicit versus explicit regularization. Distinguish explicit regularization added to a training objective from the implicit regularization observed in RL-based post-training. Explain that the latter arises from the nature of on-policy loss updates, even without explicit controls.
- RL and SFT generalization contrast. Explain how RL and SFT differed as training compute increased, including what happened to the SFT model's existing spatial reasoning in the navigation test.
- SFT as forward KL. Explain why minimizing SFT cross-entropy is equivalent to minimizing forward KL from the target distribution to the learned policy.
- RL as reverse KL. Trace how the KL-regularized RL objective corresponds to minimizing reverse KL from the learned policy to a reward-optimal policy.
- Mode-covering and mode-seeking pressures. Distinguish the optimization pressures induced by forward and reverse KL using the distribution under which each expectation is taken.
- Multimodal retention through on-policy updates. Explain why RL can forget less than SFT when adapting a pretrained policy with old and new modes, despite the initial intuition about mode covering.
- RL’s razor thesis. Explain why RL’s razor predicts less forgetting from on-policy RL: among policies that perform well on a new task, it favors those closer to the original policy in KL divergence.
- Source of smaller RL shifts. Distinguish the role of on-policy sampling from negative gradients in the reported RL–SFT comparison, and explain intuitively why on-policy updates stay nearer the model’s distribution.
- Setup-specific regularization choices. Explain why regularization methods used by leading models should not be assumed to work in every post-training setup.
- Pretraining loss during RLHF. Explain why InstructGPT adds a pretraining-style next-token prediction objective to RLHF training and what data it uses.
- Winning-completion likelihood in DPO. Distinguish the relative preference signal in the DPO loss from the added negative log-likelihood penalty on the winning completion.
- Rating-derived reward margins. Explain what the margin in Llama 2’s proposed reward-model loss represents and how annotator judgments can supply it.
What it builds on
- Quantitative over-optimization (chapter 14) · The preceding discussion of quantitative over-optimization.
- How DPO works (chapter 8) · DPO's use of winning and losing completions in preference optimization.
- Training a Bradley–Terry reward model (chapter 5) · Reward model training from comparisons.
- Direct preference optimization (chapter 8) · Direct preference optimization as a direct-alignment algorithm.
Section by section
Introduction
Introduces regularization as a way to limit harmful changes during RLHF, especially by comparing the current policy with a reference policy.
- Regularization and over-optimization: Explain why RLHF uses regularization: powerful optimization can change a model too much relative to the strong model from the previous training stage, causing performance outside the optimized setting to drop or outputs to become nonsensical.
Why it matters: It connects the need for constraints to the model failures those constraints are meant to prevent. - KL comparison with reference policy: Describe the chapter's most common regularization variant as a KL comparison between the current policy and a reference policy across generated samples, and distinguish the informal term KL distance from a true mathematical distance metric.
Why it matters: It identifies the main constraint the chapter develops without confusing an informal name with a mathematical property.
Key terms: regularization, over-optimization, reference model, Kullback–Leibler (KL) distance, KL divergence, current policy, reference policy, generated samples
15.1 KL divergence in RL optimization
Introduces KL divergence as a measure of drift from a reference policy and contrasts the two KL directions used in RL optimization.
- KL divergence and model drift: Explain what KL divergence measures and identify the two model distributions compared in RLHF.
Why it matters: This gives you the baseline for understanding what a KL penalty constrains during RLHF. - Reverse KL reference penalty: Describe how the book's common reverse-KL penalty is estimated and what kind of departure from the reference it strongly penalizes.
Why it matters: The sampling direction helps explain which changes to the RL model this penalty discourages. - Other KL direction: Distinguish the other KL direction by the probability mismatch it penalizes and its stated connection to distillation or behavioral cloning.
Why it matters: The two directions can discourage different kinds of departures from the reference.
Key terms: KL divergence, reference policy, reverse KL, KL penalty, Monte Carlo estimate, reference model, KL direction, distillation, behavioral cloning
15.1.1 Reference model to generations
Explains how a KL penalty keeps an RL-trained model’s generations close to a static reference model.
- Static reference for KL penalties: Explain that the KL penalty compares the model’s generated tokens during training with a static reference model to encourage the trained model to retain a desired style. Identify the usual reference and one alternative.
Why it matters: The choice of reference determines which model behavior the penalty encourages training to preserve.
Key terms: KL penalties, static reference model, instruction-tuned model, previous RL checkpoint
15.1.2 Implementation example
This subsection shows how to approximate KL divergence from sampled model completions and compute it from policy and reference-model log-probabilities.
- Sampled KL approximation: Explain why full sequence-level KL is impractical for LLMs and how sampling from the current model replaces a sum over possible completions with an expectation.
Why it matters: Sampling makes a KL estimate feasible without enumerating every completion. - Generated sequence log-probability comparison: Trace how the implementation uses a generated sequence to obtain policy and reference log-probabilities and approximate KL.
Why it matters: The distinction between generating tokens and scoring those same tokens explains what the implementation actually measures.
Key terms: sequence-level KL, expectation, generative distribution, reference model, generate(), forward(), per-token logits, log-probabilities, policy sequence log-probability, reference sequence log-probability
15.2 Implicit regularization
The section contrasts deliberately added regularization with the resistance to memorization and forgetting that can emerge from on-policy RL post-training.
- Implicit versus explicit regularization: Distinguish explicit regularization added to a training objective from the implicit regularization observed in RL-based post-training. Explain that the latter arises from the nature of on-policy loss updates, even without explicit controls.
Why it matters: It separates benefits that arise during on-policy RL training from controls practitioners deliberately add.
Key terms: explicit regularization, implicit regularization, on-policy optimization, memorization, catastrophic forgetting
15.2.1 SFT memorizes, RL generalizes
This section uses a controlled comparison to show that RL generalizes to changed task rules while supervised fine-tuning can improve on the training distribution yet fail under those changes.
- Out-of-distribution test shifts: Describe how the study tested whether post-training transfers beyond the training distribution, using the rule changes in GeneralPoints and V-IRL.
Why it matters: These changes make clear what the study means by transferring a rule to an unseen variant. - RL and SFT generalization contrast: Explain how RL and SFT differed as training compute increased, including what happened to the SFT model's existing spatial reasoning in the navigation test.
Why it matters: Better performance on the training distribution can conceal a serious loss of ability to handle changed rules.
Key terms: out-of-distribution (OOD) generalization, GeneralPoints, V-IRL, action space, supervised fine-tuning (SFT), RL, in-distribution, out-of-distribution (OOD) performance, spatial reasoning
15.2.2 Retaining by doing: On-policy data mitigates forgetting
The section explains why on-policy RL can retain prior knowledge better than SFT by connecting their objectives to opposite directions of KL divergence and examining how those directions affect a multimodal policy.
- SFT as forward KL: Explain why minimizing SFT cross-entropy is equivalent to minimizing forward KL from the target distribution to the learned policy.
Why it matters: This identifies the optimization pressure behind SFT’s effect on previously learned behavior. - RL as reverse KL: Trace how the KL-regularized RL objective corresponds to minimizing reverse KL from the learned policy to a reward-optimal policy.
Why it matters: The reverse-KL connection explains why RL trains on the model’s own generations rather than a fixed set of target completions. - Mode-covering and mode-seeking pressures: Distinguish the optimization pressures induced by forward and reverse KL using the distribution under which each expectation is taken.
Why it matters: The sampling direction provides the link between each mathematical objective and its behavior during training. - Multimodal retention through on-policy updates: Explain why RL can forget less than SFT when adapting a pretrained policy with old and new modes, despite the initial intuition about mode covering.
Why it matters: This resolves why mode-seeking RL can preserve prior knowledge even though mode-covering SFT might initially sound safer.
Key terms: forward KL, SFT cross-entropy loss, target distribution, negative log-likelihood, KL-regularized RL objective, reverse KL, reward-tilted distribution, partition function, reference policy, mode covering
15.2.3 RL’s razor: Why online RL forgets less
Explains RL’s razor: on-policy training tends to find high-reward policies with less KL drift from the initial model, which is associated with less forgetting.
- RL’s razor thesis: Explain why RL’s razor predicts less forgetting from on-policy RL: among policies that perform well on a new task, it favors those closer to the original policy in KL divergence.
Why it matters: It connects the kind of solution RL finds on a new task to how well the model retains past tasks. - New-task KL as forgetting indicator: Describe the reported relationship between KL drift and forgetting, including where KL is measured and why that makes the measurement useful.
Why it matters: It gives a practical way to estimate forgetting using drift measured on the new specialized data. - Source of smaller RL shifts: Distinguish the role of on-policy sampling from negative gradients in the reported RL–SFT comparison, and explain intuitively why on-policy updates stay nearer the model’s distribution.
Why it matters: It identifies the training-data distinction behind the smaller policy shifts, rather than attributing them to RL’s negative gradients.
Key terms: RL’s razor, on-policy methods, KL divergence, forgetting, new task data, initial policies, on-policy, offline data, negative gradients, SFT
15.3 Other types of regularization
The section introduces setup-specific regularization methods and cautions that examples from leading models are not universal recipes.
- Setup-specific regularization choices: Explain why regularization methods used by leading models should not be assumed to work in every post-training setup.
Why it matters: It helps you treat a model’s regularization choices as context-dependent rather than as a universal recipe.
Key terms: regularization, post-training setups, stable optimization
15.3.1 Pretraining gradients
This subsection explains how language-modeling losses can regularize RLHF and DPO training so preference optimization does not undermine text quality.
- Pretraining loss during RLHF: Explain why InstructGPT adds a pretraining-style next-token prediction objective to RLHF training and what data it uses.
Why it matters: The added objective gives the model a reason to retain language-modeling behavior while it learns from feedback. - Winning-completion likelihood in DPO: Distinguish the relative preference signal in the DPO loss from the added negative log-likelihood penalty on the winning completion.
Why it matters: A preferred answer can outrank a rejected one without being likely text in its own right.
Key terms: pretraining gradients, autoregressive next-token prediction loss, pretraining corpus, DPO logistic loss, negative log-likelihood penalty, winning completion, reference model, length-normalized
15.3.2 Margin-based regularization
This subsection introduces a rating-derived margin for reward model training and places it alongside other approaches to regularization in RLHF.
- Rating-derived reward margins: Explain what the margin in Llama 2’s proposed reward-model loss represents and how annotator judgments can supply it.
Why it matters: It shows how preference judgments can carry more information than a simple choice between two outputs. - Regularization across alignment methods: Distinguish the regularization approaches mentioned for reward models and direct-alignment algorithms, and describe where reward margins also appear.
Why it matters: It prevents treating one form of regularization as the default across every part of RLHF.
Key terms: margin loss, reward models, margin, numerical scale, quantified ranking method, Likert scales, standard contrastive loss function, KL divergence, β parameter, reward margins
15.4 Summary
Concepts explained in this chapter
- Kullback–Leibler divergenceForward KL divergence is KL divergence from a target distribution to an approximating distribution, with expectation taken under the target distribution.
- catastrophic forgettingCatastrophic forgetting is the rapid loss of previously learned capabilities or information when a model is trained on new objectives or data.
- KL regularizationA KL penalty is a regularization term proportional to the Kullback–Leibler divergence between an optimized distribution or policy and a reference distribution or policy.
- negative log-likelihoodNegative log-likelihood is the negation of a model's log-likelihood, used as a loss for probabilistic predictions.
- reverse KL divergenceReverse KL divergence is the Kullback-Leibler divergence from a target distribution to a reference distribution, penalizing probability mass assigned by the target where the reference assigns little probability.
- fine-tuningFine-tuning is further training of a pretrained model on a task-specific or behavior-specific objective.
- reinforcement learningReinforcement learning is a machine learning paradigm in which an agent learns behavior by optimizing rewards obtained through interaction with an environment.
- regularizationRegularization is a technique that constrains optimization to reduce undesirable or excessive fitting to an objective.
- reference modelA reference model is a frozen copy of a policy that provides a baseline distribution for measuring or penalizing policy deviation.
- on-policy methodOn-policy optimization, or on-policy learning, updates a policy using data generated by that same policy or its current update distribution.
- Monte Carlo simulationMonte Carlo estimation approximates an expectation or other quantity by averaging values from samples drawn from the relevant distribution.
- over-optimizationReward over-optimization is a condition in which a reward model's score continues to improve while actual quality measured by held-out evaluations or human judgment degrades.
- DPO lossThe DPO loss is a differentiable preference-training objective that increases the relative likelihood of chosen completions over rejected completions relative to a reference policy.
- preference marginA preference margin is the numerical difference in reward or rating values between two compared outputs, representing the strength of preference.
- cross entropy lossCross-entropy is the expected negative log probability assigned by one distribution to outcomes drawn from another distribution.
- mode coveringMode covering is a distribution-approximation tendency that spreads probability across the major modes of a target distribution.
- mode seekingMode seeking is a distribution-approximation tendency that concentrates probability on one or a few high-probability modes of a target distribution.
- partition functionA partition function is a normalization factor that converts unnormalized nonnegative weights into a probability distribution.
- data driftDistribution mismatch is a discrepancy between the probability distribution that generated training data and the distribution of the policy being optimized.
- log-ratioA log-ratio is the logarithm of the ratio between two positive quantities, such as model and reference probabilities.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.