Preference displacement

As explained in Reinforcement Learning from Human Feedback

Preference displacement is a shift in probability mass away from responses represented in preference-training data toward unaddressed model behaviors.

Preference displacement is a potential DPO failure mode in which optimization shifts probability mass away from both chosen and rejected responses, with the rejected response decreasing more. The proposed consequence is increased probability for unaddressed behaviors—tokens the model can generate but that preference-training data does not represent. section 8.2 section 8.6

DPO increases the margin between chosen and rejected completions rather than directly preserving the chosen response’s probability. This objective can therefore improve the relative margin while reducing both probabilities. The exact practical impact remains uncertain, and the effect is proposed as one reason online methods may outperform vanilla DPO. section 8.2 section 8.6

Mitigation strategies change either the optimization or the training data:

• Cal-DPO adjusts the optimization process, while AlphaPO modifies the reward shape. section 8.2
• Online DPO generates completions from the current model, and D2PO creates preference data through reward-model relabeling. section 8.2

Figure 8.1 Sketch of preference displacement in DPO

Use the sketch to compare how DPO changes the probabilities of chosen and rejected responses. section 8.2

How Reinforcement Learning from Human Feedback covers it

How deep it goes

Explains probability shifts and proposed causes, but stops short of establishing the effect’s exact practical impact.

Examples

Contrasts chosen and rejected response probabilities through Figure 8.1’s sketch of DPO behavior.

What it assumes

Assumes familiarity with DPO, chosen and rejected completions, preference data, and online versus offline training.

What you can do after

Diagnose probability displacement and consider optimization, reward-shaping, or online-data mitigations.

How it connects

Connects DPO and direct-alignment algorithms with offline preference data, online learning, and RLHF.

liveBookmanning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

• Direct Preference Optimizationproduced by · Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.