liveBook · Manning

Group Relative Policy Optimization

Also called GRPO

Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled actions or outputs.

How 6 Manning books explain it

Build a Reasoning Model (From Scratch)

A reinforcement-learning method that uses groups of rollouts for the same prompt and derives learning signals from their relative performance.

How it is used
Deploys verifier rewards within RLVR to update a reasoning language model from relative comparisons among grouped rollouts.
How deep it goes
Explains and implements rollout sampling, verification, advantage normalization, log probabilities, loss computation, and training, then defers extensions to later treatment.
Examples
Demonstrates MATH problems, boxed-answer verification, four-rollout rewards, zero-advantage groups, and a PyTorch RLVR training run.
What you can do after
Build it from scratch as a PyTorch RLVR training loop with grouped sampling, deterministic rewards, normalized advantages, checkpoints, and diagnostics.

Read how Build a Reasoning Model (From Scratch) explains Group Relative Policy Optimization →

Build a DeepSeek Model (From Scratch)

GRPO, or Group Relative Policy Optimization, is a reinforcement-learning algorithm that estimates relative advantages among grouped sampled responses and optimizes a policy with a clipped objective and regularization.

How it is used
Uses group-relative rewards to post-train language models toward reasoning behavior.
How deep it goes
Develops reward normalization, a clipped objective, and a training-step skeleton, while deferring distributed training and production infrastructure.
Examples
Works through four math completions with binary rewards and implements group normalization and a clipped loss in PyTorch.
What you can do after
Build a minimal GRPO training-step skeleton from scratch, with placeholders for rollout generation and token log-probability scoring.

Read how Build a DeepSeek Model (From Scratch) explains Group Relative Policy Optimization →

Applied Reinforcement Learning

A reinforcement-learning algorithm that compares rewards among multiple outputs sampled for the same prompt to estimate relative advantages for policy updates.

How it is used
Trains language models on financial reasoning tasks where answers can be checked mechanically.
How deep it goes
Works through sampling, reward normalization, clipped updates, KL monitoring, and memory-efficient training.
Examples
Evaluates a financial question-answering model on FinQA using exact-answer accuracy and boxed-answer format rates.
What you can do after
Build a GRPO training pipeline from scratch using verifiable rewards and LoRA.

Read how Applied Reinforcement Learning explains Group Relative Policy Optimization →

Reinforcement Learning from Human Feedback

GRPO is a policy-optimization algorithm that estimates relative advantages among groups of sampled outputs.

How it is used
Uses group-relative policy optimization for RLHF and reasoning tasks.
How deep it goes
Derives group-normalized advantages and a clipped token-level objective, then illustrates reward grouping and policy-loss computation.
Examples
Illustrates grouped reward statistics and clipped per-token losses with a PyTorch-style implementation.
What you can do after
Apply it via PyTorch to compute group-normalized advantages and a clipped per-token policy loss.

Read how Reinforcement Learning from Human Feedback explains Group Relative Policy Optimization →

Domain-Specific Small Language Models

A reinforcement learning optimization method that updates a policy by comparing groups of sampled actions, avoiding the need for a separate value model.

How it is used
Specializes a small language model for domain-specific reasoning using reinforcement learning.
How deep it goes
Walks through model setup, prompts, rewards, training, and inference without deriving the optimization rule.
Examples
Adapts Qwen 2.5 3B Instruct to additive-manufacturing questions on a GPU with 16 GB of VRAM.
What you can do after
Apply it via TRL’s GRPOConfig and GRPOTrainer to tune a domain-specific reasoning model with custom rewards.

Read how Domain-Specific Small Language Models explains Group Relative Policy Optimization →

LLM Customization and Fine-Tuning

A reinforcement-learning method that optimizes a policy using relative comparisons among groups of sampled outputs instead of a separate critic network.

How it is used
Uses GRPO to improve model behavior on tasks with programmatically verifiable rewards.
How deep it goes
Explains group sampling, reward normalization, policy updates, and practical compute tradeoffs.
Examples
Illustrates relative updates with four scored responses and identifies code tests, math answers, and structured extraction as reward sources.
What you can do after
Apply it via TRL by configuring a response-scoring reward function and group size.

Read how LLM Customization and Fine-Tuning explains Group Relative Policy Optimization →

Also covered in

What to know first

Related concepts

Questions about Group Relative Policy Optimization

What is Group Relative Policy Optimization?

Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled actions or outputs.

Which Manning books explain Group Relative Policy Optimization?

Build a Reasoning Model (From Scratch), Build a DeepSeek Model (From Scratch), Applied Reinforcement Learning, Reinforcement Learning from Human Feedback, Domain-Specific Small Language Models and LLM Customization and Fine-Tuning.

What should I know before learning Group Relative Policy Optimization?

Manning books that teach Group Relative Policy Optimization usually introduce Proximal Policy Optimization, reward model, reinforcement learning and reward signal first.

What is the difference between Group Relative Policy Optimization and Proximal Policy Optimization?

Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled actions or outputs. Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.

What are the parts of Group Relative Policy Optimization?

Group Relative Policy Optimization includes Kullback–Leibler divergence.

What is Group Relative Policy Optimization also called?

Group Relative Policy Optimization is also called GRPO.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.