Group Relative Policy Optimization

As explained in Reinforcement Learning from Human Feedback

GRPO is a policy-optimization algorithm that estimates relative advantages among groups of sampled outputs.

Group Relative Policy Optimization (GRPO) is a PPO-inspired policy-gradient method for RLHF that ranks several completions sampled for the same prompt. Rather than learn a value function to estimate how good each completion was, it treats the sampled group as its local comparison set: outputs rewarded above the group average receive positive advantage, while weaker outputs receive negative advantage. This makes GRPO particularly suited to settings such as reasoning training, where many candidate answers can be generated and scored for one problem. section 6.1 section 6.2.8

For a prompt s, generate G completions and assign rewards r₁ through rG. GRPO commonly computes each completion’s advantage by subtracting the group reward mean and dividing by the group standard deviation. That advantage is shared across every token in the completion. The policy update then uses a PPO-style clipped importance ratio, limiting how far the current policy can move from the rollout policy, and includes a KL penalty against a reference policy directly in the loss. section 6.2.8 section 6.3.6

Figure 6.7 GRPO architecture. Advantages are normalized relative to the group mean and standard deviation. The KL penalty is applied directly in the loss rather than shaping the reward.

The architecture highlights the training flow: sample multiple answers per prompt, normalize their rewards within each group, apply the resulting advantages to token-level policy updates, and penalize KL divergence in the loss. section 6.2.8

Compared with nearby policy-gradient methods, GRPO changes the advantage estimator rather than abandoning PPO-style control:

• PPO uses a learned value function and per-token advantage estimation; GRPO removes the value model and derives advantages from completions in the same prompt group. section 6.2.11 section 6.2.8
• RLOO also compares multiple completions, but uses the average reward of the other samples as a leave-one-out baseline; group-centered GRPO advantage is proportional to that baseline before standard-deviation normalization. section 6.2.11 section 6.2.8
• GSPO retains group-relative advantages but uses a sequence-level policy ratio to address length bias, whereas GRPO uses token-level importance sampling. section 6.2.11

A grouped rollout update

For one reasoning prompt, generate several candidate solutions and score them. A solution scoring above that prompt’s group mean receives a positive normalized advantage, so its token probabilities are increased; a below-average solution receives a negative advantage, so they are decreased. In an implementation, rewards are reshaped by prompt and number of generations, group means and standard deviations are repeated for each completion, and the resulting advantages are broadcast over token positions before clipped policy losses are accumulated. section 6.3.6

Removing the value network can reduce memory use and avoids the unresolved problem of learning a value function from a language-model backbone. The trade-off is that GRPO needs multiple generations for every prompt and depends on relative reward variation inside each group. Its clipping, loss aggregation, and KL handling remain practical stability choices rather than guarantees of universal superiority. section 6.2.8 section 6.6

How Reinforcement Learning from Human Feedback covers it

How deep it goes

Derives normalized group advantages and PPO-style clipped losses, then details reward reshaping, token losses, and KL placement.

Examples

Uses multiple scored reasoning completions per prompt and pseudocode that reshapes rewards into group-normalized token advantages.

What it assumes

Assumes familiarity with policy gradients, PPO-style clipping, token probabilities, rewards, and KL divergence.

What you can do after

Apply it via a PPO-style training pipeline that replaces a learned value model with group-normalized completion rewards.

How it connects

Connects PPO, REINFORCE, RLOO, KL divergence, importance sampling, policy gradients, and generalized advantage estimation.

liveBookmanning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Reasoning Model (From Scratch)A reinforcement-learning method that uses groups of rollouts for the same prompt and derives learning signals from their relative performance.
• 

  liveBookmanning.com
  Domain-Specific Small Language ModelsA reinforcement learning optimization method that updates a policy by comparing groups of sampled actions, avoiding the need for a separate value model.

Compare how 3 books explain Group Relative Policy Optimization →

Related concepts

• Proximal Policy Optimizationalternative to · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
• value functioncontrasts with · A value function assigns or estimates the expected cumulative reward, utility, or quality associated with a state or action under a policy or decision rule.
• KL regularizationuses · KL regularization is the use of a penalty or constraint based on Kullback–Leibler divergence to limit the difference between a learned distribution and a reference distribution.
• Kullback–Leibler divergencehas part · Kullback–Leibler divergence is a nonnegative, directed measure of the information lost when one probability distribution approximates another, computed as an expected log ratio under the first distribution.
• clipped policy ratiohas part · Clipped policy ratio is a policy-optimization technique that limits the contribution or change of new-to-old policy probability ratios to a permitted range.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.