liveBook · Manning

Process supervision

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Process supervision is a training or evaluation technique that assigns feedback to intermediate steps in a model's reasoning or task-solving process.

Process supervision assigns feedback to intermediate reasoning steps instead of judging only the final answer. It addresses credit assignment: outcome scores cannot identify where a long reasoning trace failed. section 7.6.3 section 7.7.1

A process-supervised reward pipeline typically:

  • Labels each reasoning step as correct, incorrect, or neutral using human annotation or an automated process. section 7.7.1
  • Trains a process reward model (PRM) to predict those step-level labels and score reasoning as it unfolds. section 7.7.1
  • Uses the PRM to rerank candidate responses or, in later work, as a reward signal inside reinforcement learning. section 7.7.2

Five-step math solution

An outcome reward gives one score after the final answer. A PRM can mark steps 1 and 2 correct, step 3 incorrect, and later steps misdirected, allowing a solver to prefer a sound trace even when another trace reaches the right answer accidentally. section 7.7.1 section 7.7.2

Process supervision provides denser feedback and distinguishes correct answers reached by sound reasoning from accidental success. Its main cost is labeling every step, and its advantage over verifiable outcome rewards remains unsettled outside narrow settings. section 7.7.1 section 7.7.6

Figure 7.9 MATH accuracy as a function of the number of reranked samples N, for three rerankers: PRM, ORM, and majority voting. The PRM curve sits above the ORM curve across the full range, and both sit above majority voting.

Compare PRM, ORM, and majority-vote reranking across increasing candidate counts on MATH; the PRM curve remains highest. section 7.7.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains step labels, PRM training, reranking, and tradeoffs; defers detailed training-loop implementation and broader-domain validation.
Examples
Uses five-step math traces, PRM800K, the MATH benchmark, and PRM-versus-ORM reranking results.
What it assumes
Assumes reinforcement-learning vocabulary while establishing outcome rewards, process rewards, and credit assignment through contrasts.
What you can do after
Apply process reward models to rerank candidate solutions and distinguish sound reasoning from accidental correctness.
How it connects
Contrasts outcome supervision with reward models, verifiable rewards, reinforcement learning, and credit assignment.

Also explained in

Compare how 2 books explain process supervision →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.