liveBook · Manning

Preference margin

A preference margin is the numerical difference between the reward or rating assigned to a preferred output and that assigned to a less-preferred output.

How 2 Manning books explain it

LLM Customization and Fine-Tuning

The difference between the implicit rewards assigned to a chosen response and a rejected response in a preference pair.

How it is used
Uses reward margin alongside reward accuracy to assess preference learning during DPO training and held-out evaluation.
How deep it goes
Defines the average implicit reward gap and gives interpretation ranges and troubleshooting guidance.
Examples
Reports margins of 0.09 in training and 0.05 on held-out human preference pairs.
What you can do after
Diagnose preference learning by comparing held-out margin and accuracy, then inspect the pairs if the signal remains near zero.

Read how LLM Customization and Fine-Tuning explains preference margin →

Reinforcement Learning from Human Feedback

A preference margin is the numerical difference in reward or rating values between two compared outputs, representing the strength of preference.

How it is used
Uses numerical rating gaps to preserve preference strength during reward-model training and preference optimization.
How deep it goes
Defines the rating difference, gives a worked example, and shows its loss term, but defers alternative margin functions.
Examples
Uses Likert-style ratings of 5 and 2 to calculate a margin of 3 and illustrate stronger-than-binary preference information.
What you can do after
Apply margin-aware losses and interpret how preference strength weights reward-model or alignment updates.

Read how Reinforcement Learning from Human Feedback explains preference margin →

Related concepts

Questions about preference margin

What is preference margin?

A preference margin is the numerical difference between the reward or rating assigned to a preferred output and that assigned to a less-preferred output.

Which Manning books explain preference margin?

LLM Customization and Fine-Tuning and Reinforcement Learning from Human Feedback.

What is the difference between preference margin and Llama?

A preference margin is the numerical difference between the reward or rating assigned to a preferred output and that assigned to a less-preferred output. Llama is a family of large language models developed by Meta.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.