liveBook · Manning

Process reward model

Also called PRM

A process reward model is a learned model that evaluates intermediate steps or partial trajectories in a reasoning or task process and assigns them rewards.

How 2 Manning books explain it

Reinforcement Learning from Human Feedback

A process reward model assigns rewards to intermediate steps or partial trajectories in a task or reasoning process.

How it is used
Guides reasoning evaluation, candidate reranking, and search or decoding by scoring intermediate steps.
How deep it goes
Defines step-level supervision, derives cross-entropy training, and sketches masked implementation without detailing model internals beyond the prediction head.
Examples
Uses newline-like separators, boundary labels, masked tokens, and three-class correct-neutral-incorrect predictions.
What you can do after
Build a step-scoring language-model head and apply boundary-based scores to evaluate or prune reasoning traces.

Read how Reinforcement Learning from Human Feedback explains process reward model →

LLM Evaluation and Alignment, The Foundational Ideas

A process reward model is a learned model that predicts correctness or quality labels for intermediate reasoning steps and provides step-level rewards.

How it is used
Improves credit assignment and filters reasoning candidates when intermediate steps matter.
How deep it goes
Explains step labels, model training, reranking, and reinforcement-learning use, while noting annotation cost and domain limits.
Examples
Uses five-step math reasoning, PRM800K annotations, and MATH reranking comparisons.
What you can do after
Evaluate and rerank candidate reasoning traces using step-level quality signals.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains process reward model →

What to know first

Related concepts

Questions about process reward model

What is process reward model?

A process reward model is a learned model that evaluates intermediate steps or partial trajectories in a reasoning or task process and assigns them rewards.

Which Manning books explain process reward model?

Reinforcement Learning from Human Feedback and LLM Evaluation and Alignment, The Foundational Ideas.

What should I know before learning process reward model?

Manning books that teach process reward model usually introduce language model, reward model and reinforcement learning first.

What is process reward model a kind of?

Process reward model is a kind of reward model. A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.

What is process reward model also called?

Process reward model is also called PRM.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.