An outcome reward model (ORM) scores whether a completed answer is correct. It is designed for reasoning-heavy, verifiable tasks where a candidate completion can be labeled correct or incorrect from its final outcome. Unlike a process reward model, it does not supervise or expose mistakes in intermediate reasoning: a completion receives one outcome label based only on its final answer. section 5.4 section 5.6
Given a prompt x and candidate completion y, ORM training uses correct and incorrect completions, often written (yc, yic). The model is usually a pretrained language model with a scalar prediction head, but it produces a correctness probability at each completion token rather than one sequence-level reward. Prompt tokens are masked from the loss; the binary outcome label r∈{0,1} is copied to every completion token. section 5.4 section 5.6
Equation 5.10
\[\label{eq:orm_loss} \mathcal{L}_{\text{CE}}(\theta) = -\mathbb{E}_{(s,r)\sim \mathcal{D}}[r\log p_\theta(s) + (1-r)\log(1-p_\theta(s))],\]Training minimizes token-level binary cross-entropy between the repeated completion outcome label and the predicted correctness probability. The objective operationalizes a final-answer judgment as supervision at every completion position. section 5.4
Training labels for a completed solution
For a prompt with a verifiable answer, label a correct completion 1 and an incorrect completion 0. Concatenate each prompt with its completion, assign -100 to prompt-token labels, and repeat 1 or 0 across the completion tokens. A scalar head predicts token logits; binary cross-entropy is computed only where labels are not masked. This makes the model a verifier trained from offline outcome data, not a detector of the first flawed reasoning step. section 5.4 section 5.6
At inference, an ORM takes a prompt and finished candidate completion and returns per-token probabilities pt≈P(correct at token t) over completion tokens. These must be aggregated into a response score. Common choices are the mean probability, the minimum probability for tail-risk sensitivity, the product of probabilities (or summed log-probabilities), an average over the last m tokens, or a threshold that flags any token below τ. The resulting score supports verification, filtering, reranking, and best-of-N selection. section 5.6.1 section 5.4
ORMs share a language-model backbone and small scalar head with related models, but differ in prediction target and supervision pipeline:
- A Bradley–Terry preference reward model returns one sequence-level quality score and learns from comparisons between completions; an ORM returns per-token correctness probabilities from labeled outcomes. section 5.6 section 5.6.1
- A process reward model scores intermediate reasoning steps using stepwise feedback or annotations; an ORM judges only final success or failure, even though its label is repeated per token. section 5.4 section 5.6
- A value function estimates expected remaining return from a prefix and is typically trained from changing on-policy rollout targets; an ORM estimates token-local correctness from fixed offline labels. section 5.6
Figure 5.3 Training an ORM uses offline labels from a verifier or dataset (e.g., all 1s for correct completions). Each completion token is trained with binary cross-entropy against the outcome label, and per-token probabilities are aggregated into a final score for verification, filtering, or reranking.
The training pipeline shows offline verifier or dataset labels copied across completion tokens, token-level binary cross-entropy, and aggregation into a response-level score for downstream selection. section 5.4
