Reverse KL divergence measures drift by placing the learned or current policy on the left of the KL: D_KL(πθ || πref). It is estimated from tokens sampled from that policy, then scored against a fixed reference. The loss becomes large when the current policy assigns substantial probability where the reference assigns little. section 15.1
Equation 15.3
\[\label{eq:kl_distance_regularization} \mathcal{D}_{\text{KL}}(P || Q) = \sum_{x \in \mathcal{X}} P(x) \log \left(\frac{P(x)}{Q(x)}\right).\]The KL formula makes the asymmetry operational: probabilities from the left-hand distribution weight the log-ratio, so sampling and penalty behavior depend on which policy is placed first. section 15.1
In RLHF, reverse KL is a regularizer alongside reward: maximize expected reward while penalizing D_KL(π || πref). This objective can also be written as minimizing divergence from a reward-reweighted optimal policy. The coefficient β controls the reward-versus-reference trade-off. section 15.2.2
Its practical consequences follow from sampling the policy being updated:
- It constrains generated-token updates, helping prevent excessive departure from the starting model and degenerate outputs such as repetition, language switching, or nonsensical reasoning. section 15.4
- It differs from forward KL used to characterize SFT: forward KL samples target data and stretches the policy to cover it, while reverse KL shifts probability toward the target without disturbing an old mode in the illustrated dynamics. section 15.2.2
- It supports on-policy distillation by sampling completions from a student and comparing its next-token distributions with a fixed teacher on student-induced states, avoiding offline distribution mismatch. section 12.3.2
On-policy distillation
For prompt s, sample completion a from student πθ. At each state st=(s,a<t), minimize D_KL(πθ(·|st) || πT(·|st)). The resulting signal can be expressed per sampled token as log πT(at|st) − log πθ(at|st), then optimized with an online policy-gradient framework. section 12.3.2
Equation 12.5
\[\label{eq:opd_reverse_kl} \mathcal{L}_{\mathrm{OPD}}(\theta) = \mathbb{E}_{s, a \sim \pi_\theta(\cdot \mid s)} \sum_t D_{\mathrm{KL}}\left(\pi_\theta(\cdot \mid s_t) \;\|\; \pi_T(\cdot \mid s_t)\right).\]This objective shows reverse KL applied at every token state reached by the student's own sampled completion. section 12.3.2
The same loss extends to multi-teacher on-policy distillation: weight each teacher’s reverse-KL term by a prompt-dependent mixture weight. Per-token teacher feedback commonly requires student and teacher to share a tokenizer. section 12.3.3
