Process reward model

As explained in Reinforcement Learning from Human Feedback

A process reward model assigns rewards to intermediate steps or partial trajectories in a task or reasoning process.

A process reward model (PRM) scores intermediate reasoning steps instead of assigning one reward to an entire completion. It supports evaluating whether a chain of thought is progressing soundly and can guide search by pruning weak branches. section 5.5 section 5.6.1

Given prompt x and reasoning steps s₁…sₖ, a binary PRM estimates rθ(sᵢ | x, s<ᵢ), the probability that each step is valid. Training sums cross-entropy over annotated steps, not over every token. Multiclass implementations commonly predict incorrect, neutral, or correct at designated step boundaries. section 5.5

Distinguish a PRM from nearby reward signals by examining its prediction target and aggregation:

• A standard reward model produces one sequence-level score, commonly for reranking candidates or supplying a terminal RLHF reward. section 5.6.1
• An outcome reward model scores correctness across response tokens, whereas a PRM scores reasoning steps at their boundaries. section 5.5 section 5.6.1
• A value function estimates expected remaining return from a state and supports advantage computation; it is not a probability that the current step is correct. section 5.6.1 section 5.6

Boundary-masked training pipeline

Choose a separator such as a newline, append it to each completion, and place a label only at the final token of each reasoning step. Mask all other positions with -100. A language model's hidden states feed a three-class linear head; cross-entropy is computed only where labels are not masked. At inference, aggregate boundary scores by a mean, minimum, or weighted sum over steps. section 5.5 section 5.6.1

Figure 5.5 Process RMs provide supervision only at step boundaries (e.g., newline tokens). Each step receives a 3-class label: correct (+1), neutral (0), or incorrect (-1). All other tokens are masked during training.

Use the figure to inspect how supervision appears only at reasoning-step boundaries and how correct, neutral, and incorrect labels are assigned. section 5.5

How Reinforcement Learning from Human Feedback covers it

How deep it goes

Defines step-level supervision, derives cross-entropy training, and sketches masked implementation without detailing model internals beyond the prediction head.

Examples

Uses newline-like separators, boundary labels, masked tokens, and three-class correct-neutral-incorrect predictions.

What it assumes

Assumes prompts, completions, chain-of-thought steps, pretrained language models, and basic reward-model or RLHF terminology.

What you can do after

Build a step-scoring language-model head and apply boundary-based scores to evaluate or prune reasoning traces.

How it connects

Contrasts process reward models with reward models, outcome reward models, and value functions.

liveBookmanning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

• reward modelkind of · A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
• cross entropy lossuses · Cross-entropy loss measures the expected negative log probability that a predicted distribution assigns to outcomes drawn from a target distribution.
• verifiable rewardcontrasts with · A verifiable reward is a reward signal whose correctness can be determined by an objective criterion or procedure.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.