liveBook · Manning

Reference model

As explained in Applied Reinforcement Learning

A reference model is a fixed model whose policy distribution provides a comparison point during policy training.

A reference model is a frozen policy that gives training a trusted point of comparison. A KL penalty discourages the trainable policy from drifting too far from it, helping preserve fluent, instruction-following behavior while allowing improvement. section 10.7

In PPO, the reference model is not the old policy in the update ratio: the old policy generated the current batch, while the reference model remains the frozen SFT baseline. With LoRA, disabling adapters makes the base model serve as the reference policy, avoiding a second full copy of model weights. section 10.7 section 11.3

LoRA reference pass

Disable the adapters to compute reference log-probabilities, then enable them for the policy pass. The same forward-pass infrastructure can serve both. section 11.5.2

How Applied Reinforcement Learning covers it

How deep it goes
Explains the KL penalty’s role and distinguishes the reference model from PPO’s data-generating old policy.
Examples
Uses a medical chatbot’s fluency as a behavior to preserve and shows LoRA adapters switched off for a reference pass.
What you can do after
Apply it via LoRA by disabling adapters to compute reference log-probabilities without holding a second full model copy.
How it connects
Connects reference policies to PPO, RLHF, supervised fine-tuning, KL penalties, and LoRA.

Also explained in

Compare how 4 books explain reference model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.