A reference model is a frozen copy of the policy from before reinforcement learning begins. Its token probabilities provide a stable comparison distribution, constraining optimization so reward improvements do not produce excessive drift, degraded language quality, or reward exploitation. section 7.3.1 section 7.3.3
Use the reference model as a control path during policy optimization:
- Score the same generated response tokens with both the trainable policy and the frozen reference model. section 7.12
- Compute a token-level KL penalty from the policy and reference log probabilities; larger divergence reduces the objective. section 7.3.3
- Keep the reference model separate from the value model: the former anchors distributional change, while the latter estimates expected reward. section 7.3.1
- Retain the reference-policy KL term when moving from PPO to GRPO; GRPO removes the value model, not the anchor. section 7.3.3
Reference-policy term in GRPO
For generated-token log probabilities, the objective combines clipped policy updates with reference control: L_GRPO = clipped surrogate − β KL(πθ || πref). A PyTorch implementation computes kl = logp - ref_logp and subtracts β times its mean from the surrogate before negating for minimization. Production implementations should apply response and padding masks and may use a lower-variance KL estimator. section 7.4.2 section 7.8.3
Listing 7.4 The clipped GRPO loss with reference-policy KL
def grpo_loss(
logp: torch.Tensor,
old_logp: torch.Tensor,
ref_logp: torch.Tensor,
advantages: torch.Tensor,
eps: float = 0.2,
beta: float = 0.04,
) -> torch.Tensor:
"""Compute the clipped GRPO objective as a minimization loss.
Shapes:
logp, old_logp, ref_logp: [batch_size, group_size, tokens]
advantages: [batch_size, group_size]
"""
ratio = torch.exp(logp - old_logp)
clipped = ratio.clamp(1.0 - eps, 1.0 + eps)
adv = advantages.unsqueeze(-1)
surrogate = torch.minimum(ratio * adv, clipped * adv)
kl = logp - ref_logp
return -(surrogate - beta * kl).mean()Inspect the annotated loss listing to trace current-policy, old-policy, reference-policy, advantage, clipping, and KL tensors through one implementation. section 7.8.3



