Mode covering is a distribution-approximation tendency in which a model spreads probability across the major modes of a target distribution. Forward KL encourages this behavior because it evaluates errors under the target distribution and penalizes missing regions where the target has mass. section 15.2.2
This pressure contrasts with reverse KL, which evaluates regions under the model’s own distribution and therefore tends to concentrate on one high-probability mode. In multimodal policies, that difference produces an unexpected forgetting pattern: forward KL can stretch toward a new mode and pull mass from an old one, while reverse KL can shift the new mode without disturbing the old mode. section 15.2.2
Old and new policy modes
Imagine an old mode encoding prior knowledge and a new mode representing a target task. Forward-KL training tries to cover both target modes, whereas reverse-KL training can focus on the high-reward new mode while preserving the old mode. section 15.2.2
Equation 15.5
\[\label{eq:sft_forward_kl_1} \text{KL}(\pi_\star \| \pi_\theta) = \mathbb{E}_{(x,y) \sim \mathcal{D}} \left[ \log \pi_\star(y \mid x) - \log \pi_\theta(y \mid x) \right].\]Use the forward-KL equation to inspect how expectation under the target distribution creates pressure to cover its modes. section 15.2.2
