Overfitting occurs when a model fits training data too closely instead of learning patterns that generalize. Training performance improves while held-out performance degrades, with both metrics measuring the same task on different data splits. section 14.1.1
In reward modeling, correlated comparisons from one prompt can make the model overfit individual prompts. Presenting K completions creates K-choose-2 pairs, so prompts with more completions would otherwise contribute disproportionately to training. section 5.3.2
Reduce this failure mode by grouping comparisons by their shared prompt:
- Keep all pairwise comparisons from one prompt in the same training batch. section 5.3.2
- Average the grouped comparisons so each prompt contributes one update. section 5.3.2
- Normalize the comparison loss by the number of pairs, preventing prompts with more completions from dominating optimization. section 5.3.2
Overfitting versus over-optimization
A reward model that memorizes prompt-specific comparisons exhibits overfitting. A language model that produces verbose answers or repeats rare tokens because those behaviors increase reward-model scores exhibits over-optimization instead: the proxy score improves while actual user satisfaction declines. section 5.3.2 section 14.1.1
Equation 5.8
\[\label{eq:rewardmodelinginstructgpt} \mathcal{L}(\theta) = - \frac{1}{\binom{K}{2}} \mathbb{E}_{(x, y_c, y_r)\sim D} \log \left( \sigma \left( r_{\theta}(y_c \mid x) - r_{\theta}(y_r \mid x) \right) \right).\]The pairwise reward-model objective makes the comparison normalization explicit, averaging the loss over all completion pairs associated with a prompt. section 5.3.2
