Preference margin

Q: How deeply does Reinforcement Learning from Human Feedback cover preference margin?

Defines the rating difference, gives a worked example, and shows its loss term, but defers alternative margin functions.

As explained in Reinforcement Learning from Human Feedback

A preference margin is the numerical difference in reward or rating values between two compared outputs, representing the strength of preference.

A preference margin quantifies how much more strongly one output is preferred over another. Instead of reducing ratings or rankings to only “chosen” and “rejected,” it preserves the strength of the comparison: m(y_c, y_r) = rating(y_c) − rating(y_r). section 5.3.1

In reward modeling, the margin shifts the target separation between the predicted rewards of chosen and rejected completions. The loss therefore asks the model to make rθ(y_c | x) exceed rθ(y_r | x) by an amount related to the observed preference strength. section 5.3.1 section 15.3.2

Ratings of 5 and 2

If a chosen completion receives a quality rating of 5 and a rejected completion receives 2, the preference margin is m(y_c, y_r) = 5 − 2 = 3. Larger observed gaps encode stronger preferences than simple binary labels. section 5.3.1

Equation 5.7

\[\label{eq:rewardmodelingmargin} \mathcal{L}(\theta) = - \log \left( \sigma \left( r_{\theta}(y_c \mid x) - r_{\theta}(y_r \mid x) - m(y_c, y_r) \right) \right)\]

The margin-aware logistic loss subtracts the observed margin from the predicted reward difference before applying the sigmoid and negative log. section 5.3.1

Preference margins also support broader alignment objectives:

Reward-weighted DPO, reward-aware preference optimization, and REBEL use reward differences or margins to weight updates. section 15.3.2
Direct-alignment algorithms increase the gap between chosen and rejected completion probabilities; token-level gradients reflect each token’s contribution to that gap. section 8.4
Llama 3 removed the margin term after observing diminishing improvements with scaling, so margin use remains a design choice rather than a universal standard. section 5.3.1

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

implicit rewardmeasures · An implicit reward is a reward value inferred from preferences or an optimization objective rather than supplied as an explicit score or predicted by a separately trained reward model.
Llamaused by · Llama is a family of autoregressive large language models developed by Meta and released in multiple versions and sizes.
Direct Preference Optimizationmeasured by · Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.
direct alignment algorithmproduced by · A direct alignment algorithm optimizes a model directly from preference data without learning an intermediate reward model or using a reinforcement-learning optimizer.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.