How deeply does Reinforcement Learning from Human Feedback cover on-policy distillation?

Defines student-induced states, reverse-KL loss, and token advantages, while deferring deeper reverse-KL intuition to later treatment.

On-policy distillation

As explained in Reinforcement Learning from Human Feedback

On-policy distillation is distillation in which training examples or trajectories are sampled from the current student policy and evaluated against a teacher policy.

On-policy distillation (OPD) trains a student on completions sampled from its current policy, then compares its next-token distributions with a fixed teacher. This avoids relying only on teacher-generated data, which can create a student–inference distribution mismatch and encourage overestimation of low-probability teacher regions. The resulting optimization uses a reverse-KL objective inside an online policy-gradient reinforcement-learning framework. section 12.3.2

For prompt s, completion a, and token-level state s_t = (s, a_<t), OPD proceeds as follows:

Sample a completion from the current student policy πθ(·|s), so training states reflect the behavior encountered during student inference. section 12.3.2
Compare the student distribution πθ(·|s_t) with the fixed teacher distribution πT(·|s_t) using reverse KL, DKL(πθ || πT). section 12.3.2

Equation 12.5

\[\label{eq:opd_reverse_kl} \mathcal{L}_{\mathrm{OPD}}(\theta) = \mathbb{E}_{s, a \sim \pi_\theta(\cdot \mid s)} \sum_t D_{\mathrm{KL}}\left(\pi_\theta(\cdot \mid s_t) \;\|\; \pi_T(\cdot \mid s_t)\right).\]

Formalizes the OPD loss as an expectation over student-generated trajectories and token-level reverse-KL comparisons with the teacher. section 12.3.2

Multiple specialist teachers

Multi-teacher OPD combines specialist or intermediate-checkpoint teachers, assigning prompt-dependent weights to their reverse-KL terms. Math and code teachers can therefore contribute differently across a training batch. OPD can also be combined with other advantage computations, while on-policy self-distillation lets a model verify completions itself or with external tools and train a weaker version. section 12.3.3

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

off-policy distillationcontrasts with · Off-policy distillation is knowledge distillation in which a student learns from data or outputs generated independently of its current policy.
student modelproduces · A student model is a model trained to approximate or reproduce the outputs, capabilities, or behavior of another model that provides training guidance.
distillationkind of · Knowledge distillation is a training technique in which a student model learns to reproduce or approximate the outputs or behavior of a teacher model.
reverse KL divergenceuses · Reverse KL divergence is the Kullback–Leibler divergence from an approximating distribution to a target distribution, with the expectation taken under the approximating distribution.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.