Kullback–Leibler (KL) divergence measures how much one probability distribution differs from another. In reinforcement learning from human feedback, it measures how far an RL-tuned policy has drifted from its supervised fine-tuning (SFT) reference. section 7.5.4
A KL penalty turns this comparison into a practical control loop:
- Start with the raw reward-model score for a prompt–response pair. section 7.5.4
- Subtract a small positive coefficient times the KL divergence between the current policy and the reference policy. section 7.5.4
- Optimize the resulting reward, trading reward-model improvement against policy drift. section 7.5.4
Why the penalty matters
A reward model is calibrated on SFT-model outputs. If PPO moves far from that distribution, the reward model may be unreliable and reward hacking can produce repetitive sequences or gibberish with high scores. KL discourages leaving the region where those scores are meaningful. section 7.5.4


















