KL regularization discourages a trainable policy from drifting too far from a reference policy. In PPO-based RLHF, the reference is usually a frozen supervised fine-tuned (SFT) model. The penalty lets training improve reward while anchoring behavior to that starting point. section 10.7
The KL term is subtracted from reward and weighted by β; increasing its weight makes divergence more costly. section 10.7
The KL penalty and PPO clipping stabilize training in different ways: clipping limits each update relative to the policy that generated the current data, while KL regularization limits longer-term drift from the frozen reference. Those policies are not the same. section 10.7
Anchoring a medical support chatbot
In medical-chatbot RLHF, PPO fine-tuning adds a KL penalty to the reward. The frozen SFT model acts as the reference, anchoring the policy as it learns from preference rewards and safety-related shaping. section 10.7 section 10.8
The anchor’s strength depends on the model and training behavior. A math-specialized model improved without an anchor while its KL rose slowly; for instruction-tuned models, a modest β of 0.04 kept KL in a productive range and supported steady improvement. In another PPO run, an adaptive coefficient rose from 0.1 to 10 to restrain divergence from the SFT reference. section 11.6.3 section 10.8.6




