A financial analyst may need a language model to calculate a number that can be checked against a known answer. Group Relative Policy Optimization (GRPO) trains the model by comparing several responses to the same prompt. Rather than relying on a separate value network to estimate a baseline, it uses the responses’ own rewards as the baseline. section 11.0 section 11.4
For each training prompt, GRPO follows a practical loop:
- Sample a group of responses using a temperature above zero, so the model can produce varied answers. section 11.4
- Score each response. For a task with a known answer, a verifier can assign rewards by checking correctness. section 11.0 section 11.4
- Subtract the group’s mean reward from each response’s reward and divide by the group’s reward standard deviation. The result is its relative advantage: positive above the mean and negative below it. section 11.4
- Update the policy using a PPO-style clipped objective, while monitoring divergence from a reference policy. section 11.7.5 section 11.9
Comparing answers to one prompt
Suppose four sampled answers receive correctness rewards of 1, 1, 0, and 0. Their mean is 0.5 and their population standard deviation is 0.5. The two correct answers have advantage +1; the two incorrect answers have advantage −1. If every answer receives the same reward, there is no relative advantage to distinguish them, so the group provides no advantage-driven learning signal. section 11.4
The comparison is relative to the current group, not an absolute judgment of an answer’s quality. Clipping limits the policy update, while KL divergence helps track how far the policy has moved from its reference. A useful starting model should already solve some examples, so sampled groups can contain positive rewards. section 11.4 section 11.7.5 section 11.9
For large language models, the vocabulary scores needed for token probabilities can consume substantial memory. One implementation computes selected token log-probabilities without allocating a full softmax tensor and processes responses in micro-batches, accumulating gradients before the optimizer step. section 11.5.1 section 11.7.5
Listing 11.7 GRPO steps
def grpo_step(self, batch: List[Dict]) -> Dict:
groups = []
all_rewards, all_correct, all_boxes = [], [], []
for item in batch:
roll = self.generate_group(item["prompt"])
rewards, corrects, boxes = [], [], []
for resp in roll["responses"]:
r = compute_reward(resp, item["gold"], self.config)
rewards.append(r["reward"])
corrects.append(r["correct"])
boxes.append(r["has_box"])
rewards_t = torch.tensor(
➥rewards, dtype=torch.float32, device=self.device)
advantages = (rewards_t –
➥ rewards_t.mean()) / (rewards_t.std() + 1e-8) #A
groups.append({
"full_ids": roll["full_ids"],
"response_mask": roll["response_mask"],
"advantages": advantages,
})
all_rewards.extend(rewards)
all_correct.extend(corrects)
all_boxes.extend(boxes)
for g in groups:
full_ids = g["full_ids"]
attn = (full_ids != self.tokenizer.pad_token_id).long()
with torch.no_grad():
set_lora_enabled(self.model, True)
with torch.autocast(
➥ device_type="cuda", dtype=self.model_config.dtype,
enabled=self.use_amp):
logits = self.model(input_ids=full_ids,
➥ attention_mask=attn).logits #B
g["old_logp"] = self._token_log_probs(
➥ logits, full_ids)
del logits
set_lora_enabled(self.model, False)
with torch.autocast(device_type="cuda",
➥ dtype=self.model_config.dtype,
enabled=self.use_amp):
ref_logits = self.model(input_ids=full_ids,
➥ attention_mask=attn).logits #C
g["ref_logp"] = self._token_log_probs(
➥ref_logits, full_ids)
del ref_logits
g["attn"] = attn
# STEP 4-5: Optimization.
... (see Listing 11.8)The training step shows how sampled responses are scored and normalized into group-relative advantages. section 11.7.5
In a FinQA evaluation, a GRPO-fine-tuned model improved exact-answer accuracy from 31.21% to 38.45%. A small format bonus also encouraged boxed answers, which the verifier could parse; the boxed-answer rate rose from 89.8% to 99.1%. These results illustrate both the potential of verifiable feedback and the importance of reward design. section 11.7.6
Table 11.2 Base vs. GRPO-fine-tuned accuracy on the full FinQA test set
|
Metric
|
Base model
|
GRPO model
|
Change
|
|
Exact-answer
accuracy
|
31.21%
|
38.45%
|
+7.24 pp
(+23% relative)
|
|
Boxed-answer
format rate
|
89.8%
|
99.1%
|
+9.3 pp
|
|
Trainable
parameters
|
1.58B (all)
|
36.9M
(adapters)
|
2.34% of
total
|
Compare the base and fine-tuned models’ accuracy, answer format, and trainable parameter counts. section 11.7.6





