liveBook · Manning

Reference model

As explained in LLM Customization and Fine-Tuning

A fixed policy distribution that serves as the comparison point for constraining changes to an optimized policy.

Preference training can improve a model’s chosen responses while disturbing capabilities it already has. A reference model provides a fixed comparison point: in direct preference optimization (DPO), it is a frozen copy of the pre-DPO checkpoint, while a separate policy model is updated. section 8.2.5 section 8.2.3

DPO uses chosen and rejected responses to shift the policy’s preferences relative to the reference. Its implicit reward compares the policy’s response probability with the frozen reference’s probability, scaled by beta. The reference anchors the update without requiring a separate reward model or reinforcement learning loop. section 8.2 section 8.2.5

Figure 8.3 The DPO training mechanism: raise the chosen response, lower the rejected one, stay near the frozen reference.

The diagram shows how chosen and rejected responses drive policy updates while the frozen reference constrains drift. section 8.2.5

Set the reference constraint deliberately:

  • Start with a strong supervised fine-tuning (SFT) checkpoint. A weak checkpoint becomes a weak reference, and DPO inherits its weaknesses. section 8.8.3
  • Use beta 0.1 as a baseline in TRL. Higher beta keeps updates closer to the reference; below 0.05 allows more aggressive changes but risks destabilizing other capabilities. Above 0.3, the policy may barely change. section 8.2.5 section 8.2.3
  • Budget for both model copies and their forward passes. If memory is constrained, consider reference-free ORPO or SimPO rather than assuming standard DPO is the only option. section 8.2.11

Memory planning for a 4B model

One full-parameter run needed about 54 GB for DPO, compared with about 32 GB for full SFT, because policy and reference copies were loaded together. Gradient checkpointing trades extra compute for lower memory, but this DPO run still did not fit on one 24 GB card. section 8.2.11 section 8.2.3

A reference policy also appears in reinforcement learning from human feedback (RLHF), where PPO optimizes a language model against a separately trained reward model while staying close to the reference. DPO instead uses preference pairs directly and does not train that separate reward model. section 8.4 section 8.2

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains frozen-copy behavior, implicit reward, beta thresholds, and memory tradeoffs without a full mathematical derivation.
Examples
Uses chosen and rejected response pairs and a 4B training run to illustrate policy comparison and memory overhead.
What it assumes
Assumes a completed SFT pipeline and preference pairs before DPO refines the model.
What you can do after
Apply it via TRL by choosing a strong SFT checkpoint, starting with beta 0.1, and planning for reference-model memory.
How it connects
Connects reference models to DPO, supervised fine-tuning, KL constraints, RLHF, PPO, and reward models.

Also explained in

Compare how 4 books explain reference model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.