A policy is the rule an agent uses to choose actions from states. In neural reinforcement learning, model parameters determine that rule, and training changes them to make actions associated with higher long-term reward more likely. section 7.1.1
From cleaning actions to tokens
A cleaning robot observes its room, chooses whether to move, vacuum, or return to its dock, and receives feedback as the room changes. For a language model, the policy is the probability distribution over the next token; a rollout samples tokens to form a response, which can then receive a reward. section 7.1.1 section 7.2
A basic policy-gradient update adds α times the sum over sampled tokens of R ∇θ log πθ(aₖ | sₖ) to the parameters θ. Here πθ(aₖ | sₖ) is the probability assigned to sampled token aₖ in state sₖ, α is the learning rate, and R weights the update. This compact form uses raw reward; a baseline can instead make the learning signal relative to expected reward. section 7.2.2 section 7.3.2
A clipped language-model training objective distinguishes three policy roles:
- The current policy is the trainable model. It scores generated tokens and receives the gradient update. section 7.3.1 section 7.8.4
- The old policy produced the rollout. Its stored token log probabilities are compared with current log probabilities to form the clipped probability ratio. section 7.10 section 7.8.3
- The reference policy is normally a frozen copy from before reinforcement learning. A KL penalty uses it as an anchor against excessive drift; it is distinct from a value model that predicts expected reward. section 7.3.1 section 7.12
In GRPO, sample several completions per prompt, score them, and normalize their rewards within the group to obtain relative advantages. Score the same generated tokens under the current and reference policies, then apply a clipped objective and an optimizer step. Unlike PPO's learned value baseline, the group-relative signal removes the need for a learned value model. section 7.12 section 7.8.4
Listing 7.4 The clipped GRPO loss with reference-policy KL
def grpo_loss(
logp: torch.Tensor,
old_logp: torch.Tensor,
ref_logp: torch.Tensor,
advantages: torch.Tensor,
eps: float = 0.2,
beta: float = 0.04,
) -> torch.Tensor:
"""Compute the clipped GRPO objective as a minimization loss.
Shapes:
logp, old_logp, ref_logp: [batch_size, group_size, tokens]
advantages: [batch_size, group_size]
"""
ratio = torch.exp(logp - old_logp)
clipped = ratio.clamp(1.0 - eps, 1.0 + eps)
adv = advantages.unsqueeze(-1)
surrogate = torch.minimum(ratio * adv, clipped * adv)
kl = logp - ref_logp
return -(surrogate - beta * kl).mean()The simplified PyTorch loss shows the current-to-old probability ratio, clipping, group advantages, and reference-policy KL term. A real trainer must also mask prompt and padding tokens; the displayed KL estimate is simplified. section 7.8.3
For a real trainer, store generated token IDs and old log probabilities during rollout. When rescoring, gather each generated token's log probability from the preceding model position, and mask prompt and padding tokens out of the loss. Monitor reward alongside KL: rising reward with exploding KL is a warning sign. section 7.10 section 7.12




















