liveBook · Manning

Reward accuracy

As explained in LLM Customization and Fine-Tuning

The proportion of preference pairs for which a model assigns a higher implicit reward to the chosen response than to the rejected response.

Reward accuracy measures the proportion of preference pairs for which a model assigns a higher implicit reward to the chosen response than to the rejected response. In direct preference optimization (DPO), it shows how often the model ranks responses in the preferred direction—not how large the score gap is. section 8.2.3 section 8.2.4

Read reward accuracy alongside reward margin, the average score gap between chosen and rejected responses. Both should rise as the model learns. Prefer held-out accuracy when assessing generalization: training accuracy can reflect both ongoing adjustment and memorization of training pairs. section 8.2.3 section 8.2.4

A DPO training run

A run on 270 training pairs reached about 73% training reward accuracy, up from roughly 45%. On 30 held-out pairs, accuracy was 60%, modestly above the 50% chance line. The training and held-out margins were 0.09 and 0.05, respectively: positive but small gaps on contested human preferences. section 8.2.3 section 8.2.4

Table 8.3 DPO training summary for the reference run
Metric
Value
Training loss
0.653
Training reward accuracy
73%
Training reward margin
0.09
Eval loss
0.673
Eval reward accuracy
60%
Eval reward margin
0.05
Training pairs
270
Validation pairs
30
Training time
~3.5 minutes (sharded across three 24 GB GPUs)
Starting checkpoint
SFT model from chapter 6

Compare accuracy with reward margin and loss for the training and held-out pairs. section 8.2.4

Use the metrics to decide what to investigate next:

  • If held-out accuracy is above chance and the margin is positive, evaluate the model further rather than treating the training figure as the final result. section 8.2.3 section 8.2.4
  • If accuracy stays near chance or the margin near zero, inspect and fix the preference pairs before adjusting beta or the learning rate. section 8.2.4
  • Treat near-100% held-out accuracy on contested human preferences as a warning that the task may be too easy or leaky, not automatically as success. section 8.2.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines the pairwise proportion and gives interpretation and warning guidance alongside reward margin.
Examples
Reports 73% training and 60% held-out accuracy for a run using 270 training and 30 validation pairs.
What you can do after
Diagnose/debug with reward accuracy and margin to identify preference pairs that may need fixing before hyperparameter changes.
How it connects
Connects direct preference optimization with reward margin, loss, and held-out evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.