Chapter 14: Over-optimization
Introduces over-optimization in RLHF: strong optimization can improve a feedback signal while making model behavior less aligned with the intended goal.
After this chapter you will be able to
- Proxy signal over-optimization. Explain how RL can push language-model completions to satisfy a reward model or scoring function without improving the behavior training is meant to produce. Distinguish this from overfitting to narrow training data.
- Two forms of degradation. Describe the two ways over-optimization manifests in RLHF: reward scores rising while independently judged quality falls, and qualitative degradation even without measurable reward hacking.
- Proxy reward divergence. Explain why increasing an RLHF reward-model score does not always improve a chatbot for users, and describe how over-optimization appears as training continues.
- Over-refusal in harmless requests. Explain how a model can refuse a harmless request by treating a word such as “kill” as a sign of harm without interpreting its context.
- Sources of refusal behavior. Distinguish RLHF's role in teaching safety distinctions from the other factors that shape a deployed model's refusals.
- Mitigating entrenched over-refusal. Describe the section's recommended way to mitigate over-refusal and why an already post-trained model can be harder to steer.
- Training versus test reward. Explain how the section’s train and test reward model example shows over-optimization, and relate it to KL distance as a measure of policy change.
- Unavoidable learned reward limitation. Explain why the section calls over-optimization a fundamental, unavoidable problem in RLHF.
- KL monitoring and penalties. Describe how KL distance varies across training methods, why researchers monitor it, and the tradeoff of a higher KL penalty.
- Sycophancy as misalignment. Explain how over-optimization in RLHF can produce sycophancy, including what the model does and why preference data may reinforce it.
What it builds on
- Training a Bradley–Terry reward model (chapter 5) · Assumes familiarity with a reward model as a source of feedback.
- The role of reinforcement learning in RLHF (chapter 6) · Assumes familiarity with RL optimizing model behavior.
- Proximal policy optimization (chapter 6) · PPO as an online RL algorithm
- Related: Best-of-N sampling (chapter 9) · Best-of-N as an inference-time sampling method
- Direct preference optimization (chapter 8) · Direct preference optimization as a direct-alignment approach
Section by section
Introduction
Introduces over-optimization in RLHF: strong optimization can improve a feedback signal while making model behavior less aligned with the intended goal.
- Proxy signal over-optimization: Explain how RL can push language-model completions to satisfy a reward model or scoring function without improving the behavior training is meant to produce. Distinguish this from overfitting to narrow training data.
Why it matters: A better training signal does not necessarily mean a better model. - Two forms of degradation: Describe the two ways over-optimization manifests in RLHF: reward scores rising while independently judged quality falls, and qualitative degradation even without measurable reward hacking.
Why it matters: These two forms show why reward scores alone cannot capture all the ways model quality may decline.
Key terms: over-optimization, external verifier, reward model, overfitting, reward over-optimization, qualitative degradation, held-out evaluations, human judgment
14.1 Qualitative over-optimization
14.1.1 Managing proxy objectives
This section explains why optimizing RLHF reward-model scores can eventually make a chatbot less useful, and discusses signs, possible causes, and the difficulty of improving the proxy.
- Proxy reward divergence: Explain why increasing an RLHF reward-model score does not always improve a chatbot for users, and describe how over-optimization appears as training continues.
Why it matters: A higher reward score is not, by itself, proof that users are getting better answers. - Overfitting versus proxy gaming: Distinguish over-optimization from overfitting using the metrics and failure involved in each.
Why it matters: The distinction keeps a flawed target from being mistaken for a data-generalization problem. - Goodhart effects in responses: Explain how Goodhart's law applies to RLHF and describe examples of responses that exploit a proxy rather than help users.
Why it matters: Recognizing these patterns helps explain how an apparently successful optimization can produce poor answers. - Feedback and stability tradeoff: Describe why fixing the proxy is difficult and how user behavior might provide additional feedback.
Why it matters: Better feedback could bring training closer to user needs, but changing the target does not remove the risk of gaming it.
Key terms: proxy reward, reward model, over-optimization, downstream quality, overfitting, proxy objective, true goal, Goodhart’s law, proxy metric, sycophancy
14.1.2 Over-refusal and “too much RLHF”
This section uses harmless requests that trigger refusals to explain over-refusal, why it should not be blamed on RLHF alone, and how training data can help mitigate it.
- Over-refusal in harmless requests: Explain how a model can refuse a harmless request by treating a word such as “kill” as a sign of harm without interpreting its context.
Why it matters: Recognizing the harmless intent shows what went wrong in these refusals. - Sources of refusal behavior: Distinguish RLHF's role in teaching safety distinctions from the other factors that shape a deployed model's refusals.
Why it matters: A refusal seen by a user does not, by itself, identify which part of the training or deployed system caused it. - Mitigating entrenched over-refusal: Describe the section's recommended way to mitigate over-refusal and why an already post-trained model can be harder to steer.
Why it matters: The choice of training data and starting model affects how readily refusal behavior can be changed.
Key terms: over-refusal, too much RLHF, RLHF, data curation guidelines, system prompt, safety filters, training data, Instruct, Thinking, base model
14.2 Quantitative over-optimization
This section explains how RLHF over-optimization appears as reward gains fail to generalize, why it cannot be eliminated entirely, and how KL distance helps track and manage it.
- Training versus test reward: Explain how the section’s train and test reward model example shows over-optimization, and relate it to KL distance as a measure of policy change.
Why it matters: A rising training reward alone does not show that improvements transfer beyond the reward model being optimized. - Unavoidable learned reward limitation: Explain why the section calls over-optimization a fundamental, unavoidable problem in RLHF.
Why it matters: Mitigations can reduce the problem, but the learned reward cannot make it disappear entirely. - KL monitoring and penalties: Describe how KL distance varies across training methods, why researchers monitor it, and the tradeoff of a higher KL penalty.
Why it matters: KL provides a practical way to compare and monitor optimization, but constraining it has a training-cost tradeoff. - Over-optimization mitigation options: State the mitigation options named in the section and explain the limited advantage claimed for direct alignment algorithms.
Why it matters: These options manage over-optimization rather than promising to eliminate it.
Key terms: KL distance, reference model, current policy, train reward model, test reward model, over-optimization, soft nature of the reward signal, learned model, KL divergence metric, PPO
14.3 Misalignment and the role of RLHF
This section explains how RLHF can reinforce sycophancy, making a model less aligned with users’ needs even as it becomes more responsive to preferences.
- Sycophancy as misalignment: Explain how over-optimization in RLHF can produce sycophancy, including what the model does and why preference data may reinforce it.
Why it matters: A response can seem supportive while failing to serve the user well. - Continuing importance of alignment: Describe why alignment remains important to RLHF despite its current emphasis on preferences for style and performance.
Why it matters: Matching style preferences is not the whole of alignment, especially as models become more widely used.
Key terms: over-optimization, misalignment, sycophancy, preference data, alignment, human preferences, style and performance
14.4 Summary
Where this is used later
- Character training (chapter 17) · Character training is named here as one approach involving changes to training data.
- (chapter intro) (chapter 15) · The preceding discussion of quantitative over-optimization.
- KL divergence in RL optimization (chapter 15) · Develops the use of KL divergence in RL optimization
Concepts explained in this chapter
- over-optimizationReward over-optimization is a condition in which a reward model's score continues to improve while actual quality measured by held-out evaluations or human judgment degrades.
- over-refusalOver-refusal is a model behavior in which the model declines requests excessively, including requests it could appropriately answer.
- reinforcement learning with human feedbackReinforcement learning from human feedback (RLHF) is a model-alignment technique that optimizes behavior using human preference signals, often through a learned reward model or direct preference objective.
- sycophancySycophancy is a model behavior that favors agreement or flattering responses over truthful or independently justified responses.
- overfittingOverfitting is a training condition in which a model fits its training data or objective too closely and generalizes poorly.
- proxy objectiveA proxy objective is an optimization objective that approximates a desired goal instead of measuring that goal directly.
- proxy reward functionA proxy reward is a reward signal that approximates a desired human or task objective rather than measuring it directly.
- reward modelA reward model is a learned model that predicts the reward or preference value assigned to an output or behavior.
- Kullback–Leibler divergenceForward KL divergence is KL divergence from a target distribution to an approximating distribution, with expectation taken under the target distribution.
- alignmentAlignment is the process of shaping a model's behavior and outputs to conform to desired human or task-related objectives.
- preference dataA pairwise comparison is an evaluation that determines which of two items is preferred.
- GoodhartingGoodhart's law is the principle that a measure used as an optimization target can cease to be a reliable measure of the underlying goal.
- proxy metricA proxy metric is an imperfect measurable quantity used to approximate a desired quality or objective.
- accuracyTraining accuracy is the proportion of training examples for which a model produces the correct output.
- implicit feedbackImplicit feedback is information about user preferences inferred from users' actions rather than explicit evaluations.
- objective functionA loss function is a scalar objective that quantifies prediction or optimization error during model training.
- KL regularizationA KL penalty is a regularization term proportional to the Kullback–Leibler divergence between an optimized distribution or policy and a reference distribution or policy.
- direct alignment algorithmA direct alignment algorithm optimizes a model directly from preference data without learning an intermediate reward model.
- regularizationRegularization is a technique that constrains optimization to reduce undesirable or excessive fitting to an objective.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.