A log probability is the logarithm of the probability assigned to an event. In language-model reinforcement learning, it records how strongly a policy assigned a generated token in its state. The gradient of this value turns reward into pressure to increase or decrease that token's probability. section 7.2.1 section 7.2.2
Use token log probabilities to connect sampled completions with policy updates:
- Record old-policy log probabilities during rollout; the clipped ratio compares the updated policy with the policy that generated the response. section 7.10 section 7.12
- Run current and reference policies on prompt-plus-completion sequences, gather probabilities for generated tokens, and mask prompt and padding tokens. section 7.10 section 7.8.3
- Apply the completion's advantage across its generated tokens; clipping and reference-policy KL constrain excessively large changes. section 7.9.1
Sampling and reinforcement
After a partial answer, a policy might assign probabilities 0.42 to “has,” 0.23 to “owns,” and 0.11 to “will.” Sampling can select a lower-probability token, whose log probability is then weighted by the completion reward or advantage. section 7.2.1 section 7.9.1
Listing 7.4 The clipped GRPO loss with reference-policy KL
def grpo_loss(
logp: torch.Tensor,
old_logp: torch.Tensor,
ref_logp: torch.Tensor,
advantages: torch.Tensor,
eps: float = 0.2,
beta: float = 0.04,
) -> torch.Tensor:
"""Compute the clipped GRPO objective as a minimization loss.
Shapes:
logp, old_logp, ref_logp: [batch_size, group_size, tokens]
advantages: [batch_size, group_size]
"""
ratio = torch.exp(logp - old_logp)
clipped = ratio.clamp(1.0 - eps, 1.0 + eps)
adv = advantages.unsqueeze(-1)
surrogate = torch.minimum(ratio * adv, clipped * adv)
kl = logp - ref_logp
return -(surrogate - beta * kl).mean()The clipped GRPO listing exposes the implementation path: exponentiate the difference between current and old log probabilities, clip the ratio, weight by advantages, and subtract reference-policy KL. section 7.8.3









