A process reward model (PRM) predicts whether each intermediate reasoning step is correct, incorrect, or neutral. Unlike an outcome reward model, which assigns one score to a final answer, a PRM supplies dense feedback as reasoning unfolds and helps locate failures. section 7.7.1
A practical PRM workflow is:
- Label reasoning traces step by step using positive, neutral, or negative judgments. section 7.7.2
- Train a transformer reward model with a scalar output head for step-level classification. section 7.7.2
- Score candidate solutions during inference and rerank them by process quality; later work also uses PRMs as rewards inside reinforcement-learning loops. section 7.7.2
Five-step math solution
For a five-step solution, an outcome reward only reports that the final answer is bad. A PRM can mark steps 1 and 2 correct, identify step 3 as wrong, and expose why later steps failed. On MATH, PRM reranking reached about 78% accuracy at large candidate-pool sizes, compared with 72% for ORM reranking and 65% for majority voting. section 7.7.1 section 7.7.2
Figure 7.9 MATH accuracy as a function of the number of reranked samples N, for three rerankers: PRM, ORM, and majority voting. The PRM curve sits above the ORM curve across the full range, and both sit above majority voting.
Compare PRM, outcome-model, and majority-voting reranking as candidate count increases. section 7.7.2

