Mode seeking

As explained in Reinforcement Learning from Human Feedback

Mode seeking is a distribution-approximation tendency that concentrates probability on one or a few high-probability modes of a target distribution.

Mode seeking concentrates probability on one or a few high-probability modes instead of covering every mode of a target distribution. Reverse KL encourages this behavior because it evaluates regions sampled from the model itself. section 15.2.2

Reverse-KL mode seeking has two operational pressures:

• It can ignore target regions where the model assigns almost no probability, even when the target has substantial mass there. section 15.2.2
• It penalizes probability placed outside the target and concentrates on a high-probability or high-reward region. section 15.2.2
• In multimodal policies, this concentration can preserve an old mode while shifting a sampled new mode toward the target. section 15.2.2

Old and new policy modes

Suppose a policy contains an old mode representing prior knowledge and a new mode for a target task. Forward-KL SFT stretches the policy across both modes and can pull mass away from the old one. Reverse-KL RL can move the new mode toward the target without disturbing the old mode. section 15.2.2

Figure 15.1 Forgetting dynamics for forward KL (SFT) versus reverse KL (RL). The “old” mode represents prior knowledge, and the “new” mode represents the target task. Forward KL stretches the policy to cover the target and pulls mass away from the old mode (top right), whereas reverse KL shifts the new mode toward the target without disturbing the old mode (bottom right). From Chen et al. (2025), with permission of the author.

Compare the visualized forgetting dynamics of forward-KL SFT and reverse-KL RL across old and new modes. section 15.2.2

How Reinforcement Learning from Human Feedback covers it

How deep it goes

Contrasts forward and reverse KL expectations, then explains their effects on multimodal policy forgetting.

Examples

Uses a two-mode policy whose old prior-knowledge mode coexists with a new target-task mode.

What it assumes

Assumes familiarity with probability distributions, KL divergence, supervised fine-tuning, and reinforcement learning objectives.

What you can do after

Evaluate how KL direction changes mode coverage, concentration, and preservation of prior policy behavior.

How it connects

Links reverse KL with reinforcement learning, forward KL, supervised fine-tuning, mode covering, multimodality, and catastrophic forgetting.

liveBookmanning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

• reverse KL divergenceproduced by · Reverse KL divergence is the Kullback–Leibler divergence from an approximating distribution to a target distribution, with the expectation taken under the approximating distribution.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.