A reference model is a frozen policy distribution used as the baseline during alignment training. RLHF starts from a capable pretrained or instruction-tuned model rather than learning language from scratch; optimization should improve rewarded behavior without losing that prior behavior. The reference model anchors that starting point while the trainable policy changes. section 6.1 section 3.1.4
The standard control is a Kullback–Leibler (KL) penalty between the current policy πRL and reference policy πref. For a prompt x and completion y, training maximizes reward while subtracting λKL DKL(πRL(y|x) || πref(y|x)). Operationally, the reward model supplies a reason to change; the reference model prices how far that change moves the output distribution from the baseline. section 15.1.1 section 15.0
Equation 3.11
\[\label{eq:rlhf_opt_eq} {\max_\pi \; \mathbb{E}_{\tau \sim \pi} \left[r_\theta(s_t, a_t)\right] - \beta \mathcal{D}_{\text{KL}}(\pi_{\text{RL}}(\cdot|s_t) \| \pi_{\text{ref}}(\cdot|s_t)).}\]The objective makes the trade-off explicit: maximize learned reward, but pay a β-weighted penalty whenever the RL policy diverges from the frozen reference distribution. section 3.1.4
Using a reference model requires three distinct choices:
- Freeze it. It is a copy of the policy before RL and receives no gradient updates; otherwise the baseline would move with the policy being constrained. section 6.1
- Choose the anchor checkpoint. The usual reference is the instruction-tuned model, though a previous RL checkpoint can serve instead when that is the behavior to preserve. section 15.1.1
- Set the penalty strength. A larger β keeps the policy closer to its starting point; a smaller β permits more reward-seeking deviation, consuming more of the informal KL budget. section 3.1.4 section 3.3
Preventing reward-driven drift
Suppose reward optimization increasingly favors a completion pattern that scores well but harms general behavior. Its token probabilities can move far from those of the instruction-tuned reference. The KL term lowers the effective reward for that deviation, creating pressure to retain the reference model’s style and capabilities. This regularization addresses over-optimization, which can appear as repeated text, language switching, excessive special characters, or plausible-looking but incorrect reasoning. section 15.1.1 section 15.0
A reference model is not the same as PPO’s old policy. PPO records the log-probabilities from the policy that generated the current batch and uses them in probability ratios and clipping to limit a single update. The model frozen at the start of training instead supplies the KL constraint across training. Keeping these roles separate avoids treating batch-time sampling data as the long-term behavioral anchor. section 6.3.5
The same fixed-baseline idea also appears in direct preference optimization: log-likelihood ratios compare a trainable policy with a fixed reference, often the SFT checkpoint. With an added negative-log-likelihood term, preference optimization can be balanced against maintaining high likelihood on preferred text. Thus, the reference model is a general alignment baseline, not only an RLHF implementation detail. section 15.3.1
