Outcome-only supervision trains or evaluates a model with one scalar judgment of its final result. It can identify that an answer is bad, but not where a reasoning trace failed. Outcome reward models (ORMs) apply this pattern at the end of a response. section 7.7.1
Use the contrast with process supervision to identify what the signal can and cannot support:
- Outcome supervision scores only the final answer, making it cheaper than labeling every intermediate reasoning step. section 7.7.1
- Process supervision scores each step as correct, incorrect, or neutral, providing denser feedback for credit assignment. section 7.7.1
- Outcome supervision cannot distinguish a correct answer reached through faulty reasoning from one reached through correct reasoning. section 7.7.1
- Longer responses worsen its credit-assignment problem, while reward hacking and other RLHF failure modes remain possible. section 7.8
Five-step math problem
An outcome reward assigns one score after the final answer. If that answer is wrong, the model learns only that the complete response failed. A process reward model can mark steps 1 and 2 correct, identify step 3 as the error, and expose why later steps went wrong. section 7.7.1
