A preference margin is the implicit reward assigned to a chosen response minus the reward assigned to a rejected response. Reported reward margins average this gap across preference pairs. A positive average indicates that preferred responses score higher overall; its size indicates how far apart the scores are. section 8.2.4 section 8.2.3
Read the margin alongside reward accuracy, which measures how often the chosen response scores higher. Accuracy captures the frequency of correct rankings; the margin captures the size of the score gap. Favor held-out results over training results when assessing whether the preference signal generalizes. section 8.2.3 section 8.2.4
Reading a DPO run
A run trained for one epoch on 270 preference pairs and evaluated on 30 held-out pairs. Its training reward accuracy was 73%, with a margin of 0.09; held-out accuracy was 60%, with a margin of 0.05. The small positive held-out margin suggests modest separation on contested human preference pairs, not a large or clean gap. section 8.2.3 section 8.2.4
Use the margin to decide what to inspect next:
- Compare results with the difficulty of the pairs. Margins near or above 1.0 can indicate cleanly separable pairs, often synthetic; small positive margins can reflect a weak but real signal on difficult human data. section 8.2.4 section 8.2.3
- If accuracy stays near chance or the margin stays near zero, inspect and fix the preference pairs before adjusting beta or the learning rate. section 8.2.4

