A reference model is a frozen policy that gives training a trusted point of comparison. A KL penalty discourages the trainable policy from drifting too far from it, helping preserve fluent, instruction-following behavior while allowing improvement. section 10.7
In PPO, the reference model is not the old policy in the update ratio: the old policy generated the current batch, while the reference model remains the frozen SFT baseline. With LoRA, disabling adapters makes the base model serve as the reference policy, avoiding a second full copy of model weights. section 10.7 section 11.3
LoRA reference pass
Disable the adapters to compute reference log-probabilities, then enable them for the policy pass. The same forward-pass infrastructure can serve both. section 11.5.2



