liveBook · Manning

Preference margin

As explained in LLM Customization and Fine-Tuning

The difference between the implicit rewards assigned to a chosen response and a rejected response in a preference pair.

A preference margin is the implicit reward assigned to a chosen response minus the reward assigned to a rejected response. Reported reward margins average this gap across preference pairs. A positive average indicates that preferred responses score higher overall; its size indicates how far apart the scores are. section 8.2.4 section 8.2.3

Read the margin alongside reward accuracy, which measures how often the chosen response scores higher. Accuracy captures the frequency of correct rankings; the margin captures the size of the score gap. Favor held-out results over training results when assessing whether the preference signal generalizes. section 8.2.3 section 8.2.4

Reading a DPO run

A run trained for one epoch on 270 preference pairs and evaluated on 30 held-out pairs. Its training reward accuracy was 73%, with a margin of 0.09; held-out accuracy was 60%, with a margin of 0.05. The small positive held-out margin suggests modest separation on contested human preference pairs, not a large or clean gap. section 8.2.3 section 8.2.4

Use the margin to decide what to inspect next:

  • Compare results with the difficulty of the pairs. Margins near or above 1.0 can indicate cleanly separable pairs, often synthetic; small positive margins can reflect a weak but real signal on difficult human data. section 8.2.4 section 8.2.3
  • If accuracy stays near chance or the margin stays near zero, inspect and fix the preference pairs before adjusting beta or the learning rate. section 8.2.4

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines the average implicit reward gap and gives interpretation ranges and troubleshooting guidance.
Examples
Reports margins of 0.09 in training and 0.05 on held-out human preference pairs.
What you can do after
Diagnose preference learning by comparing held-out margin and accuracy, then inspect the pairs if the signal remains near zero.
How it connects
Connects implicit rewards with reward accuracy, DPO loss, and preference pairs.

Also explained in

Compare how 2 books explain preference margin →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.