Process supervision assigns feedback to intermediate reasoning steps instead of judging only the final answer. It addresses credit assignment: outcome scores cannot identify where a long reasoning trace failed. section 7.6.3 section 7.7.1
A process-supervised reward pipeline typically:
- Labels each reasoning step as correct, incorrect, or neutral using human annotation or an automated process. section 7.7.1
- Trains a process reward model (PRM) to predict those step-level labels and score reasoning as it unfolds. section 7.7.1
- Uses the PRM to rerank candidate responses or, in later work, as a reward signal inside reinforcement learning. section 7.7.2
Five-step math solution
An outcome reward gives one score after the final answer. A PRM can mark steps 1 and 2 correct, step 3 incorrect, and later steps misdirected, allowing a solver to prefer a sound trace even when another trace reaches the right answer accidentally. section 7.7.1 section 7.7.2
Process supervision provides denser feedback and distinguishes correct answers reached by sound reasoning from accidental success. Its main cost is labeling every step, and its advantage over verifiable outcome rewards remains unsettled outside narrow settings. section 7.7.1 section 7.7.6
Figure 7.9 MATH accuracy as a function of the number of reranked samples N, for three rerankers: PRM, ORM, and majority voting. The PRM curve sits above the ORM curve across the full range, and both sit above majority voting.
Compare PRM, ORM, and majority-vote reranking across increasing candidate counts on MATH; the PRM curve remains highest. section 7.7.2

