Kullback–Leibler (KL) divergence measures how one probability distribution differs from another. For distributions P and Q, KL(P || Q) is the expected log-probability difference log P(x) − log Q(x), with samples drawn from P. KL is commonly called a distance in optimization, but it is not a true distance metric. section 15.1 section 15.0
Equation 15.3
\[\label{eq:kl_distance_regularization} \mathcal{D}_{\text{KL}}(P || Q) = \sum_{x \in \mathcal{X}} P(x) \log \left(\frac{P(x)}{Q(x)}\right).\]The discrete form weights each log-ratio by P(x), so the order of P and Q determines both the sampling distribution and the penalty behavior. section 15.1
Forward KL places the target distribution first: KL(P || Q), where P is the target and Q is the learned approximation. Its expectation is taken under the target, so optimization is driven by target-distributed samples. In supervised fine-tuning, P is the training-data distribution and Q is the policy being trained; minimizing the objective is equivalent to minimizing forward KL up to the target entropy constant. section 15.2.2 section 12.3.1
KL direction changes what an objective emphasizes:
- Forward KL samples from the target distribution and penalizes a learned policy for failing to assign probability to target-likely regions. section 15.2.2
- Reverse KL swaps the order, sampling from the learned policy; common RLHF penalties compare the current policy against a reference policy in this direction. section 15.1
- A reverse-KL penalty becomes especially large when the current policy assigns substantial probability where the reference assigns low probability. section 15.1
Teacher-to-student distillation
With a fixed teacher distribution q and student distribution p, cross-entropy H(q, p) decomposes into teacher entropy H(q) plus KL(q || p). Since the teacher entropy is constant, minimizing cross-entropy minimizes forward KL from teacher to student. Offline distillation and SFT-like training use this direction; sampling instead from the student yields a reverse-KL on-policy objective. section 12.3.1 section 12.3.2
In RLHF, KL serves a different operational role: regularizing reward maximization against a starting reference policy. The objective subtracts β KL(πRL || πref); larger β keeps the policy closer to the reference, while smaller β permits more reward-seeking change. Teams treat the resulting divergence as a KL budget and monitor it because very large values can signal a broken run or increased over-optimization risk. section 3.1.4 section 14.2
Equation 15.4
\[\label{eq:kl_expectation} \mathcal{D}_{\text{KL}}(P || Q) = \mathbb{E}_{x \sim P} \left[ \log P(x) - \log Q(x) \right].\]For language models, the expectation form enables a sample-based approximation: generate from the current policy, then compare its sequence log-probability with the reference model’s. section 15.1.2
A full sequence-level calculation is intractable because it sums over all possible completions. A practical approximation autoregressively generates a sequence, obtains per-token log-probabilities from the policy and frozen reference model, sums them across the sequence, and subtracts reference log-probability from policy log-probability. section 15.1.2
KL regularization does not eliminate over-optimization. As training continues, reward-model gains can stop transferring to held-out preference evaluation, and downstream quality can eventually decline. Larger policy models, reward-model ensembles, optimizer changes, and improved data are among the mitigations alongside KL penalties. section 14.2 section 14.4
