How deeply does Reinforcement Learning from Human Feedback cover off-policy distillation?

Derives sequence and token objectives through cross-entropy and forward KL, but stops short of online student-policy sampling.

Off-policy distillation

As explained in Reinforcement Learning from Human Feedback

Offline knowledge distillation trains a student on data or outputs generated in advance by a teacher rather than by the current student policy.

Off-policy distillation trains a student from teacher-generated outputs prepared before student optimization. The student therefore learns from a fixed dataset rather than from sequences sampled from its current policy. section 12.3.1

For sequence distillation, matching the teacher distribution over every possible sequence is intractable. A practical approximation replaces that distribution with one high-probability sequence, generated by teacher beam search, and trains the student with token-level negative log-likelihood. section 12.3.1

The technique has three operational properties:

Generates training targets a priori, commonly using teacher-produced sequences rather than the student’s own completions. section 12.3.1
Optimizes cross-entropy from the teacher distribution to the student; with a fixed teacher, this is equivalent to minimizing forward KL divergence. section 12.3.1
Trades cheaper, simpler training data generation for distribution mismatch when student outputs differ from teacher-generated sequences at inference time. section 12.3.2

Beam-search sequence distillation

Given a prompt s, beam search produces a teacher completion û. Instead of summing over every possible completion, training minimizes −log p(û | s), implemented as the sum of token-level losses along û. section 12.3.1

Equation 12.2

\[\label{eq:sequence_kd} \begin{aligned}[b] \mathcal{L}_{\mathrm{SEQ-KD}}(s) &= -\sum_{u \in \mathcal{U}} q(u \mid s)\log p(u \mid s) \notag\\ &\approx -\log p(\hat{u} \mid s) \notag\\ \end{aligned}\]

Formalizes sequence distillation as an intractable teacher expectation approximated by the negative log-likelihood of one beam-search output. section 12.3.1

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

on-policy distillationcontrasts with · On-policy distillation is a knowledge-distillation technique in which a student learns from teacher feedback on examples or trajectories sampled from the student's own policy.
Kullback–Leibler divergenceuses · Kullback–Leibler divergence is a nonnegative, directed measure of the information lost when one probability distribution approximates another, computed as an expected log ratio under the first distribution.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.