liveBook · Manning

Reference model

As explained in Build a DeepSeek Model (From Scratch)

A reference policy is a fixed or separately maintained policy whose output probabilities provide a comparison distribution during policy optimization.

A reference model is a frozen copy of the policy from before reinforcement learning begins. Its token probabilities provide a stable comparison distribution, constraining optimization so reward improvements do not produce excessive drift, degraded language quality, or reward exploitation. section 7.3.1 section 7.3.3

Use the reference model as a control path during policy optimization:

  • Score the same generated response tokens with both the trainable policy and the frozen reference model. section 7.12
  • Compute a token-level KL penalty from the policy and reference log probabilities; larger divergence reduces the objective. section 7.3.3
  • Keep the reference model separate from the value model: the former anchors distributional change, while the latter estimates expected reward. section 7.3.1
  • Retain the reference-policy KL term when moving from PPO to GRPO; GRPO removes the value model, not the anchor. section 7.3.3

Reference-policy term in GRPO

For generated-token log probabilities, the objective combines clipped policy updates with reference control: L_GRPO = clipped surrogate − β KL(πθ || πref). A PyTorch implementation computes kl = logp - ref_logp and subtracts β times its mean from the surrogate before negating for minimization. Production implementations should apply response and padding masks and may use a lower-variance KL estimator. section 7.4.2 section 7.8.3

Listing 7.4 The clipped GRPO loss with reference-policy KL
def grpo_loss(
    logp: torch.Tensor,
    old_logp: torch.Tensor,
    ref_logp: torch.Tensor,
    advantages: torch.Tensor,
    eps: float = 0.2,
    beta: float = 0.04,
) -> torch.Tensor:
    """Compute the clipped GRPO objective as a minimization loss.
 
    Shapes:
        logp, old_logp, ref_logp: [batch_size, group_size, tokens]
        advantages: [batch_size, group_size]
    """
    ratio = torch.exp(logp - old_logp)
    clipped = ratio.clamp(1.0 - eps, 1.0 + eps)
    adv = advantages.unsqueeze(-1)
    surrogate = torch.minimum(ratio * adv, clipped * adv)
    kl = logp - ref_logp
    return -(surrogate - beta * kl).mean()

Inspect the annotated loss listing to trace current-policy, old-policy, reference-policy, advantage, clipping, and KL tensors through one implementation. section 7.8.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains freezing, matched token scoring, KL penalties, and implementation caveats, while deferring complete production trainer details.
Examples
Uses a clipped GRPO equation, a PyTorch loss listing, and a training-step skeleton that scores completions under both policies.
What it assumes
Assumes policy-gradient and language-model log-probability basics while distinguishing the value model from zero.
What you can do after
Operate/configure a frozen reference policy and include its token-level KL term in a GRPO-style update.
How it connects
Connects policy optimization to PPO, GRPO, KL divergence, value models, reward models, and verifiers.

Also explained in

Compare how 4 books explain reference model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.