liveBook · Manning

Group Relative Policy Optimization

As explained in Applied Reinforcement Learning

A reinforcement-learning algorithm that compares rewards among multiple outputs sampled for the same prompt to estimate relative advantages for policy updates.

A financial analyst may need a language model to calculate a number that can be checked against a known answer. Group Relative Policy Optimization (GRPO) trains the model by comparing several responses to the same prompt. Rather than relying on a separate value network to estimate a baseline, it uses the responses’ own rewards as the baseline. section 11.0 section 11.4

For each training prompt, GRPO follows a practical loop:

  • Sample a group of responses using a temperature above zero, so the model can produce varied answers. section 11.4
  • Score each response. For a task with a known answer, a verifier can assign rewards by checking correctness. section 11.0 section 11.4
  • Subtract the group’s mean reward from each response’s reward and divide by the group’s reward standard deviation. The result is its relative advantage: positive above the mean and negative below it. section 11.4
  • Update the policy using a PPO-style clipped objective, while monitoring divergence from a reference policy. section 11.7.5 section 11.9

Comparing answers to one prompt

Suppose four sampled answers receive correctness rewards of 1, 1, 0, and 0. Their mean is 0.5 and their population standard deviation is 0.5. The two correct answers have advantage +1; the two incorrect answers have advantage −1. If every answer receives the same reward, there is no relative advantage to distinguish them, so the group provides no advantage-driven learning signal. section 11.4

The comparison is relative to the current group, not an absolute judgment of an answer’s quality. Clipping limits the policy update, while KL divergence helps track how far the policy has moved from its reference. A useful starting model should already solve some examples, so sampled groups can contain positive rewards. section 11.4 section 11.7.5 section 11.9

For large language models, the vocabulary scores needed for token probabilities can consume substantial memory. One implementation computes selected token log-probabilities without allocating a full softmax tensor and processes responses in micro-batches, accumulating gradients before the optimizer step. section 11.5.1 section 11.7.5

Listing 11.7 GRPO steps
def grpo_step(self, batch: List[Dict]) -> Dict:
    groups = []
    all_rewards, all_correct, all_boxes = [], [], []
 
    for item in batch:
        roll = self.generate_group(item["prompt"])
        rewards, corrects, boxes = [], [], []
        for resp in roll["responses"]:
            r = compute_reward(resp, item["gold"], self.config)
            rewards.append(r["reward"])
            corrects.append(r["correct"])
            boxes.append(r["has_box"])
 
        rewards_t = torch.tensor(
        ➥rewards, dtype=torch.float32, device=self.device)
 
        advantages = (rewards_t –
           ➥ rewards_t.mean()) / (rewards_t.std() + 1e-8)    #A
 
        groups.append({
            "full_ids":      roll["full_ids"],
            "response_mask": roll["response_mask"],
            "advantages":    advantages,
        })
        all_rewards.extend(rewards)
        all_correct.extend(corrects)
        all_boxes.extend(boxes)
 
    for g in groups:
        full_ids = g["full_ids"]
        attn = (full_ids != self.tokenizer.pad_token_id).long()
 
        with torch.no_grad():
            set_lora_enabled(self.model, True)
            with torch.autocast(
        ➥ device_type="cuda", dtype=self.model_config.dtype,
                                enabled=self.use_amp):
                logits = self.model(input_ids=full_ids, 
                ➥ attention_mask=attn).logits    #B
            g["old_logp"] = self._token_log_probs(
            ➥ logits, full_ids)
            del logits
 
            set_lora_enabled(self.model, False)
            with torch.autocast(device_type="cuda", 
            ➥ dtype=self.model_config.dtype,
                                enabled=self.use_amp):
                ref_logits = self.model(input_ids=full_ids, 
                ➥ attention_mask=attn).logits    #C
            g["ref_logp"] = self._token_log_probs(
            ➥ref_logits, full_ids)
            del ref_logits
        g["attn"] = attn
 
    # STEP 4-5: Optimization.
    ... (see Listing 11.8)

The training step shows how sampled responses are scored and normalized into group-relative advantages. section 11.7.5

In a FinQA evaluation, a GRPO-fine-tuned model improved exact-answer accuracy from 31.21% to 38.45%. A small format bonus also encouraged boxed answers, which the verifier could parse; the boxed-answer rate rose from 89.8% to 99.1%. These results illustrate both the potential of verifiable feedback and the importance of reward design. section 11.7.6

Table 11.2 Base vs. GRPO-fine-tuned accuracy on the full FinQA test set
Metric
Base model
GRPO model
Change
Exact-answer accuracy
31.21%
38.45%
+7.24 pp (+23% relative)
Boxed-answer format rate
89.8%
99.1%
+9.3 pp
Trainable parameters
1.58B (all)
36.9M (adapters)
2.34% of total

Compare the base and fine-tuned models’ accuracy, answer format, and trainable parameter counts. section 11.7.6

How Applied Reinforcement Learning covers it

How deep it goes
Works through sampling, reward normalization, clipped updates, KL monitoring, and memory-efficient training.
Examples
Evaluates a financial question-answering model on FinQA using exact-answer accuracy and boxed-answer format rates.
What you can do after
Build a GRPO training pipeline from scratch using verifiable rewards and LoRA.
How it connects
Connects GRPO to proximal policy optimization, reinforcement learning with verifiable rewards, LoRA, and KL divergence.

Also explained in

Compare how 6 books explain Group Relative Policy Optimization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.