An on-policy method updates a policy from outputs generated by that policy’s current version, or from a closely related checkpoint. The central benefit is alignment between the data being scored and the distribution the model actually produces: optimization acts on its own likely completions rather than on a fixed external corpus. section 6.0 section 11.2
In RLHF, the pipeline is prompt → current policy completion → reward-model score → policy-gradient update. A frozen reference model may also score the completion to supply a KL penalty, limiting excessive drift while the reward signal improves behavior. Strict policy-gradient derivations assume every rollout comes from the latest policy before its update. section 6.0 section 6.3.3
Figure 6.1 Overview of the RLHF training loop. A prompt from the dataset is passed to the tuned policy, which generates a completion. The reward model scores this completion, and the frozen initial model (typically the instruction-tuned model before RL) computes log probabilities on the same text to calculate a Kullback–Leibler penalty that prevents excessive drift. The combined reward signal then drives an RL update to the policy parameters.
Trace the RLHF loop from a prompt through generation, reward scoring, KL control, and the resulting policy update. section 6.0
Exact freshness is costly in production. Generation and learning commonly run on separate actor and learner GPUs; weights move from learners to actors while both sides continue working. This makes rollouts only nearly on-policy, but avoids idle compute and is treated as a practical stability trade-off, especially when long reasoning rollouts make inference the bottleneck. section 6.3.3
On-policy data plays related but distinct roles across post-training:
- For RL, it supplies the model-generated completions that reward-guided policy-gradient updates optimize. section 6.0
- For preference collection, it means gathering comparisons from the current model family or a preceding checkpoint, so labels cover relevant generation patterns. section 11.2 section 11.5
- For direct-alignment algorithms, including some completions from the fine-tuned model can target the token space its contrastive loss must optimize. section 8.4
- For regularization, sampling outputs already assigned non-negligible probability constrains updates near the current distribution and is associated with reduced forgetting relative to offline data. section 15.2.3
Checkpoint-matched preference pairs
Start from a supervised-fine-tuned checkpoint, generate two answers to a prompt from that checkpoint or its close relatives, and collect a preference label. The resulting pair is more useful for the next RLHF stage than a pooled completion dataset when it reflects the errors and alternatives the training policy actually produces. section 11.2 section 11.5
On-policy optimization is not a guarantee of success. Human preference collection remains expensive and operationally difficult, and preference labels can transfer biases such as verbosity, sycophancy, prefix, or formatting preferences. In distributed RL, increasing asynchrony also raises the challenge of preserving a stable learning signal despite stale rollouts. section 11.5 section 6.3.3
The broader practical implication is that data provenance becomes part of the optimization design. Modern multistage recipes combine instruction tuning, on-policy preference data, and reinforcement learning; on-policy sampling can also act as implicit regularization by favoring high-reward solutions closer in KL divergence to the starting policy. section 3.3 section 15.4
