A policy optimized against a reward model can drift toward responses that score highly but do not reflect the intended behavior. KL regularization limits that drift by penalizing differences between the current policy and a reference policy. Kullback–Leibler (KL) divergence measures the difference between their response distributions. section 7.5.4 section 7.5.1
In reinforcement learning from human feedback (RLHF), the reference is often the supervised fine-tuned (SFT) model that also initializes the policy. The optimization trades reward-model gains against the cost of moving away from that reference: adjusted reward = reward-model score − β × KL divergence. The positive coefficient β controls the penalty; one cited implementation uses about 0.02. section 7.5.1 section 7.5.4
Figure 7.8 The InstructGPT three-stage pipeline. Stage 1 fine-tunes the pretrained model on labeler-written demonstrations. Stage 2 trains a reward model on labeler-ranked comparisons of model outputs. Stage 3 optimizes the SFT model with reinforcement learning using PPO, with the reward model's score (penalized by KL divergence from the SFT model) as the reward signal. The SFT model from stage 1 plays two roles: it initializes the RL policy and is the reference distribution in the KL penalty. Source: Figure 2 of Ouyang et al. 2022.
The pipeline shows how the SFT model supplies both the initial policy and the reference for the KL penalty. section 7.5.1
KL regularization and PPO clipping serve different roles. PPO clips individual policy updates to keep the updated policy close to the previous one; the KL penalty discourages drift from the SFT reference. Both help limit unstable optimization, but neither makes the reward model a perfect measure of quality. section 7.5.4 section 7.8
A guardrail with a tradeoff
A reward model may overvalue numbered lists or hedging, leading the policy to use them even when they do not improve an answer. A KL penalty can restrain this overoptimization, but it also limits useful behavioral changes. Track reward-model scores against held-out human preferences: rising scores alongside flat or falling human ratings signal possible reward hacking. section 7.6.1
KL regularization also appears with group relative policy optimization (GRPO) and reference-based statistical rewards. In that setting, constrained updates are part of a broader setup that includes a strong base model, curated prompts, and high-quality reference answers; the penalty alone does not make a fixed metric reliable. section 7.7.4




