liveBook · Manning

Process reward model

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A process reward model is a learned model that predicts correctness or quality labels for intermediate reasoning steps and provides step-level rewards.

A process reward model (PRM) predicts whether each intermediate reasoning step is correct, incorrect, or neutral. Unlike an outcome reward model, which assigns one score to a final answer, a PRM supplies dense feedback as reasoning unfolds and helps locate failures. section 7.7.1

A practical PRM workflow is:

  • Label reasoning traces step by step using positive, neutral, or negative judgments. section 7.7.2
  • Train a transformer reward model with a scalar output head for step-level classification. section 7.7.2
  • Score candidate solutions during inference and rerank them by process quality; later work also uses PRMs as rewards inside reinforcement-learning loops. section 7.7.2

Five-step math solution

For a five-step solution, an outcome reward only reports that the final answer is bad. A PRM can mark steps 1 and 2 correct, identify step 3 as wrong, and expose why later steps failed. On MATH, PRM reranking reached about 78% accuracy at large candidate-pool sizes, compared with 72% for ORM reranking and 65% for majority voting. section 7.7.1 section 7.7.2

Figure 7.9 MATH accuracy as a function of the number of reranked samples N, for three rerankers: PRM, ORM, and majority voting. The PRM curve sits above the ORM curve across the full range, and both sit above majority voting.

Compare PRM, outcome-model, and majority-voting reranking as candidate count increases. section 7.7.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains step labels, model training, reranking, and reinforcement-learning use, while noting annotation cost and domain limits.
Examples
Uses five-step math reasoning, PRM800K annotations, and MATH reranking comparisons.
What it assumes
Assumes reasoning traces and outcome rewards, then distinguishes their step-level and final-answer supervision.
What you can do after
Evaluate and rerank candidate reasoning traces using step-level quality signals.
How it connects
Connects process supervision with outcome reward models, reinforcement learning, candidate reranking, and verifiable rewards.

Also explained in

Compare how 2 books explain process reward model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.