Reward accuracy measures the proportion of preference pairs for which a model assigns a higher implicit reward to the chosen response than to the rejected response. In direct preference optimization (DPO), it shows how often the model ranks responses in the preferred direction—not how large the score gap is. section 8.2.3 section 8.2.4
Read reward accuracy alongside reward margin, the average score gap between chosen and rejected responses. Both should rise as the model learns. Prefer held-out accuracy when assessing generalization: training accuracy can reflect both ongoing adjustment and memorization of training pairs. section 8.2.3 section 8.2.4
A DPO training run
A run on 270 training pairs reached about 73% training reward accuracy, up from roughly 45%. On 30 held-out pairs, accuracy was 60%, modestly above the 50% chance line. The training and held-out margins were 0.09 and 0.05, respectively: positive but small gaps on contested human preferences. section 8.2.3 section 8.2.4
Table 8.3 DPO training summary for the reference run
|
Metric
|
Value
|
|
Training loss
|
0.653
|
|
Training reward accuracy
|
73%
|
|
Training reward margin
|
0.09
|
|
Eval loss
|
0.673
|
|
Eval reward accuracy
|
60%
|
|
Eval reward margin
|
0.05
|
|
Training pairs
|
270
|
|
Validation pairs
|
30
|
|
Training time
|
~3.5 minutes (sharded across three 24 GB GPUs)
|
|
Starting checkpoint
|
SFT model from chapter 6
|
Compare accuracy with reward margin and loss for the training and held-out pairs. section 8.2.4
Use the metrics to decide what to investigate next:
- If held-out accuracy is above chance and the margin is positive, evaluate the model further rather than treating the training figure as the final result. section 8.2.3 section 8.2.4
- If accuracy stays near chance or the margin near zero, inspect and fix the preference pairs before adjusting beta or the learning rate. section 8.2.4
- Treat near-100% held-out accuracy on contested human preferences as a warning that the task may be too easy or leaky, not automatically as success. section 8.2.3
