Regularization constrains RLHF optimization so a capable pretrained model can improve on a reward signal without drifting into over-optimization. The practical target is not maximum reward alone: excessive optimization can reduce out-of-distribution performance and produce repeated text, language switching, incorrect reasoning, or nonsensical outputs. section 15.0 section 1.2
The standard mechanism subtracts a KL-divergence penalty between the current policy and a fixed reference policy from reward. The coefficient β sets the trade-off: larger values preserve the starting policy more strongly, while smaller values permit more reward-seeking change. Teams can treat the resulting divergence as a KL budget to spend during fine-tuning. section 3.1.4 section 15.4
Equation 3.11
\[\label{eq:rlhf_opt_eq} {\max_\pi \; \mathbb{E}_{\tau \sim \pi} \left[r_\theta(s_t, a_t)\right] - \beta \mathcal{D}_{\text{KL}}(\pi_{\text{RL}}(\cdot|s_t) \| \pi_{\text{ref}}(\cdot|s_t)).}\]The objective makes the constraint operational: maximize expected reward while paying a β-weighted cost for divergence from the reference policy. section 3.1.4
Regularization can act through several parts of the post-training pipeline:
- Use a KL penalty on generated samples to discourage probability mass that the reference assigns low probability. section 15.4
- Mix pretraining gradients into RLHF, or add length-normalized negative log-likelihood on preferred completions in DPO, to retain absolute language-modeling quality. section 15.3.1
- Apply margin losses in reward-model training when annotation ratings provide a quantified difference between responses. section 15.3.2
- Use PPO clipping as a separate step-size constraint; monitor clip fraction as a proxy for when its clipping mechanism may be active. section 6.4.2
PPO-native and reference-distance regularization are distinct. With one gradient step per batch, common in language-model training, PPO clipping may never engage, leaving KL penalties as the dominant explicit constraint. More update iterations can make clipping relevant again, but stability and training configuration determine the balance. section 6.4.2
Diagnosing the optimization trade-off
Track downstream performance against KL distance from the reference model. Performance may improve initially as the policy follows the reward, then degrade as distance grows; use that curve to identify over-optimization rather than treating a higher reward as sufficient evidence of a better model. section 14.4 section 15.0
Not every safeguard is explicit. On-policy RL can implicitly resist memorization and catastrophic forgetting, and its sampling process biases optimization toward solutions near the base model. Explicit penalties remain useful when stronger, directly tunable control over policy drift is needed. section 15.2 section 15.4
