liveBook · Manning

Group Relative Policy Optimization

As explained in LLM Customization and Fine-Tuning

A reinforcement-learning method that optimizes a policy using relative comparisons among groups of sampled outputs instead of a separate critic network.

Group Relative Policy Optimization (GRPO) trains a policy by comparing responses sampled for the same prompt. It replaces Proximal Policy Optimization’s (PPO’s) separate critic network with a baseline derived from the group’s rewards, reducing memory use while retaining a reinforcement-learning approach. section 8.3 section 8.3.1

GRPO is most useful when correctness can be checked automatically, such as with code tests, math answers, or extracted fields matched against ground truth. A reward function and prompts can supply the training signal without pre-collected human preference pairs. section 8.3

A training step uses the group as its own baseline:

  • Sample a prompt and generate a group of responses from the current policy, typically 8 to 16. section 8.3.1
  • Score each response, then calculate its advantage as its reward minus the group mean, divided by the group standard deviation. section 8.3.1
  • Update the policy toward above-average responses and away from below-average ones, with a KL penalty against a reference policy. section 8.3.1

Comparing four responses

Suppose four responses score 0.85, 0.20, 0.78, and 0.10. Their mean is approximately 0.48, so the update favors the first and third responses and moves away from the other two. section 8.3.1

Figure 8.5 One GRPO training step: sample a group of G responses, score them, normalize within the group, update toward the above-average ones.

Follow one training step from response sampling through reward normalization and the policy update. section 8.3.1

The critic savings do not make generation free. Training must generate G responses per prompt, so inference compute scales linearly with group size: G=16 costs roughly 16 times the inference compute of one generation per step. A cheap string match or regex reward is easier to afford than a judge model that performs inference for every response. section 8.3.3

Choose the approach against the task and budget:

  • Check supervised fine-tuning data quality first. For subjective quality or style refinement, start with Direct Preference Optimization (DPO); move to GRPO when verifiable rewards justify the added compute. section 8.7
  • Use GRPO when sampling multiple responses provides useful exploration beyond fixed preference pairs. Under certain conditions, group size two makes GRPO mathematically equivalent to DPO, so GRPO may offer little difference on a DPO-suited subjective task. section 8.3
  • Run safety regression tests after preference optimization: improving helpfulness can weaken refusal boundaries. section 8.10

TRL provides a GRPO implementation. Set the group size, supply a callable that scores individual responses, and allocate GPU memory for generating the group. The DeepSeek-R1 cold-start pattern combines light supervised fine-tuning for output format with large-scale GRPO to incentivize reasoning. section 8.3.3 section 8.10

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains group sampling, reward normalization, policy updates, and practical compute tradeoffs.
Examples
Illustrates relative updates with four scored responses and identifies code tests, math answers, and structured extraction as reward sources.
What it assumes
Requires prompts, a response-scoring reward function, and GPU capacity to generate groups during training.
What you can do after
Apply it via TRL by configuring a response-scoring reward function and group size.
How it connects
Compares GRPO with PPO and DPO, including conditional equivalence to DPO for groups of two.

Also explained in

Compare how 6 books explain Group Relative Policy Optimization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.