Group Relative Policy Optimization
Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled outputs or actions.
Build a Reasoning Model (From Scratch)
A reinforcement-learning method that uses groups of rollouts for the same prompt and derives learning signals from their relative performance.
- How it is used
- Deploys verifier rewards within RLVR to update a reasoning language model from relative comparisons among grouped rollouts.
- How deep it goes
- Explains and implements rollout sampling, verification, advantage normalization, log probabilities, loss computation, and training, then defers extensions to later treatment.
- Examples
- Demonstrates MATH problems, boxed-answer verification, four-rollout rewards, zero-advantage groups, and a PyTorch RLVR training run.
- What you can do after
- Build it from scratch as a PyTorch RLVR training loop with grouped sampling, deterministic rewards, normalized advantages, checkpoints, and diagnostics.
Read how Build a Reasoning Model (From Scratch) explains Group Relative Policy Optimization →
Reinforcement Learning from Human Feedback
GRPO is a policy-optimization algorithm that estimates relative advantages among groups of sampled outputs.
- How it is used
- Optimizes RLHF policies by ranking multiple completions from each prompt, especially for reasoning tasks.
- How deep it goes
- Derives normalized group advantages and PPO-style clipped losses, then details reward reshaping, token losses, and KL placement.
- Examples
- Uses multiple scored reasoning completions per prompt and pseudocode that reshapes rewards into group-normalized token advantages.
- What you can do after
- Apply it via a PPO-style training pipeline that replaces a learned value model with group-normalized completion rewards.
Read how Reinforcement Learning from Human Feedback explains Group Relative Policy Optimization →
Domain-Specific Small Language Models
A reinforcement learning optimization method that updates a policy by comparing groups of sampled actions, avoiding the need for a separate value model.
- How it is used
- Applies reinforcement learning to specialize small language models for domain-specific reasoning with constrained compute.
- How deep it goes
- Explains the PPO contrast, reward design, configuration, training, saving, and evaluation, but does not derive the optimization mathematically.
- Examples
- Uses additive-manufacturing data, XML-formatted reasoning prompts, custom reward functions, and a Qwen 2.5 3B Instruct model.
- What you can do after
- Builds a domain-specialized reasoning model using TRL’s GRPOConfig and GRPOTrainer on consumer-GPU-class hardware.
Read how Domain-Specific Small Language Models explains Group Relative Policy Optimization →
Related concepts
- Proximal Policy Optimizationalternative to · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
- value functioncontrasts with · A value function assigns or estimates the expected cumulative reward, utility, or quality associated with a state or action under a policy or decision rule.
- KL regularizationuses · KL regularization is the use of a penalty or constraint based on Kullback–Leibler divergence to limit the difference between a learned distribution and a reference distribution.
- Kullback–Leibler divergencehas part · Kullback–Leibler divergence is a nonnegative, directed measure of the information lost when one probability distribution approximates another, computed as an expected log ratio under the first distribution.
- clipped policy ratiohas part · Clipped policy ratio is a policy-optimization technique that limits the contribution or change of new-to-old policy probability ratios to a permitted range.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.