Process reward model
A process reward model is a learned model that evaluates intermediate steps or partial trajectories in a reasoning or task process and assigns them rewards.
Reinforcement Learning from Human Feedback
A process reward model assigns rewards to intermediate steps or partial trajectories in a task or reasoning process.
- How it is used
- Guides reasoning evaluation, candidate reranking, and search or decoding by scoring intermediate steps.
- How deep it goes
- Defines step-level supervision, derives cross-entropy training, and sketches masked implementation without detailing model internals beyond the prediction head.
- Examples
- Uses newline-like separators, boundary labels, masked tokens, and three-class correct-neutral-incorrect predictions.
- What you can do after
- Build a step-scoring language-model head and apply boundary-based scores to evaluate or prune reasoning traces.
Read how Reinforcement Learning from Human Feedback explains process reward model →
LLM Evaluation and Alignment, The Foundational Ideas
A process reward model is a learned model that predicts correctness or quality labels for intermediate reasoning steps and provides step-level rewards.
- How it is used
- Improves credit assignment and filters reasoning candidates when intermediate steps matter.
- How deep it goes
- Explains step labels, model training, reranking, and reinforcement-learning use, while noting annotation cost and domain limits.
- Examples
- Uses five-step math reasoning, PRM800K annotations, and MATH reranking comparisons.
- What you can do after
- Evaluate and rerank candidate reasoning traces using step-level quality signals.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains process reward model →
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- reward modelA reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent learns behavior or a decision policy through interaction with an environment and evaluative feedback, such as rewards.
Related concepts
- reward modelkind of · A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
- cross entropy lossuses · Cross-entropy loss measures the expected negative log probability that a predicted distribution assigns to outcomes drawn from a target distribution.
- credit assignmentaddresses · Credit assignment is the problem of determining which actions, decisions, or components contributed to an observed outcome, reward, or error.
- language modeling headuses · A language modeling head is a model output component that maps hidden representations to scores or probabilities over a vocabulary of tokens.
- process supervisionused by · Process supervision trains or evaluates a system by providing feedback on intermediate steps in its reasoning or task-solving process.
Questions about process reward model
What is process reward model?
A process reward model is a learned model that evaluates intermediate steps or partial trajectories in a reasoning or task process and assigns them rewards.
Which Manning books explain process reward model?
Reinforcement Learning from Human Feedback and LLM Evaluation and Alignment, The Foundational Ideas.
What should I know before learning process reward model?
Manning books that teach process reward model usually introduce language model, reward model and reinforcement learning first.
What is process reward model a kind of?
Process reward model is a kind of reward model. A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
What is process reward model also called?
Process reward model is also called PRM.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

