Online DPO

As explained in Reinforcement Learning from Human Feedback

Online DPO is a DPO variant that samples new completions from the current model during training and incorporates preference signals at training time.

Online DPO addresses a limitation of vanilla DPO: its preference signal comes from completions produced by previous or other models. It samples generations from the current model and incorporates preference signals during training, keeping the optimization loop closer to the model’s evolving behavior. section 8.2

Online preference-optimization variants differ in how they refresh training data:

• Online DPO samples new generations from the current model. section 8.2
• D2PO uses reward-model relabeling to create preference data on the fly. section 8.2
• These online approaches may outperform vanilla DPO, although their exact impact remains uncertain. section 8.2

Figure 8.1 Sketch of preference displacement in DPO

Use the figure to inspect how preference displacement can reduce both chosen and rejected response probabilities, with the rejected response reduced more strongly. section 8.2

How Reinforcement Learning from Human Feedback covers it

How deep it goes

Defines the data-generation distinction from vanilla DPO but does not specify an implementation procedure or optimization objective.

Examples

Uses current-model generations, D2PO relabeling, and a figure illustrating preference displacement.

How it connects

Connects DPO with RLHF, reward models, D2PO, Cal-DPO, AlphaPO, and preference displacement.

liveBookmanning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.