The Bradley-Terry model converts a pairwise judgment into a probability. Each item has a positive latent strength p, so the probability that i is preferred to j is p_i/(p_i+p_j). For preference learning, strengths are usually written as p_i=e^{r_i}, giving P(i>j)=σ(r_i-r_j). A larger score gap means a stronger predicted preference. section 5.1
Equation 5.2
\[\label{eq:bradterry_unbounded} {P(i > j) = \frac{e^{r_i}}{e^{r_i} + e^{r_j}} = \sigma(r_i-r_j).}\]The score form makes the central property explicit: predictions depend on the difference r_i-r_j, not on either absolute score. section 5.1
This relative nature has an important consequence: adding the same constant to every score leaves every pairwise probability unchanged. The model therefore learns an ordering and confidence in pairwise comparisons rather than a uniquely calibrated absolute reward scale. It is a useful approximation to noisy human preferences, not a claim that human judgment follows this form exactly. section 5.1
In RLHF reward modeling, a language model is converted into a scalar scorer, commonly by attaching a small linear head to its final hidden state. Given a prompt x and two completions, the reward model produces r_θ(y_1|x) and r_θ(y_2|x); their difference becomes the predicted probability that one completion is chosen over the other. section 5.1 section 5.9
Training on a chosen and rejected completion
For a labeled pair y_c ≻ y_r, maximize σ(r_θ(y_c|x)-r_θ(y_r|x)), or equivalently minimize −log σ(r_θ(y_c|x)-r_θ(y_r|x)). The loss decreases when the chosen completion receives a higher score and grows when the rejected completion scores higher. An equivalent softplus form is log(1+e^{r_θ(y_r|x)-r_θ(y_c|x)}). section 5.1
Equation 5.5
\[\label{eq:rewardmodeling1} \mathcal{L}(\theta) = - \log \left( \sigma \left( r_{\theta}(y_c \mid x) - r_{\theta}(y_r \mid x) \right) \right).\]The negative log-likelihood objective directly trains the score difference required by the Bradley-Terry preference probability. section 5.1
The model supplies a shared pairwise-preference structure for several alignment pipelines:
- Reward modeling compresses human judgments into scalar rewards that downstream RLHF optimization can use. section 5.9
- DPO starts from pairwise preference data modeled with Bradley-Terry probabilities, then expresses preferences through policy-to-reference likelihood ratios rather than a separately trained reward model. section 8.1.2
- The DPO objective retains the same log-sigmoid preference-likelihood pattern while using chosen and rejected completions under a policy and reference policy. section 8.1.2
A standard reward model is only one preference-modeling choice. Outcome reward models predict per-token correctness, process reward models score reasoning steps, and research variants include margin losses, K-wise ranking models, and aspect-conditioned reward models. No single variant is identified as universal best practice. section 5.9
