Reference model
A reference model is a fixed policy model whose output distribution provides a baseline for comparing or constraining an optimized policy.
Applied Reinforcement Learning
A reference model is a fixed model whose policy distribution provides a comparison point during policy training.
- How it is used
- Constrains policy training by penalizing divergence from a trusted frozen starting policy.
- How deep it goes
- Explains the KL penalty’s role and distinguishes the reference model from PPO’s data-generating old policy.
- Examples
- Uses a medical chatbot’s fluency as a behavior to preserve and shows LoRA adapters switched off for a reference pass.
- What you can do after
- Apply it via LoRA by disabling adapters to compute reference log-probabilities without holding a second full model copy.
Read how Applied Reinforcement Learning explains reference model →
Build a DeepSeek Model (From Scratch)
A reference policy is a fixed or separately maintained policy whose output probabilities provide a comparison distribution during policy optimization.
- How it is used
- Anchors PPO- and GRPO-style policy optimization against excessive distributional drift.
- How deep it goes
- Explains freezing, matched token scoring, KL penalties, and implementation caveats, while deferring complete production trainer details.
- Examples
- Uses a clipped GRPO equation, a PyTorch loss listing, and a training-step skeleton that scores completions under both policies.
- What you can do after
- Operate/configure a frozen reference policy and include its token-level KL term in a GRPO-style update.
Read how Build a DeepSeek Model (From Scratch) explains reference model →
LLM Customization and Fine-Tuning
A fixed policy distribution that serves as the comparison point for constraining changes to an optimized policy.
- How it is used
- Anchors preference optimization to a frozen pre-DPO checkpoint so policy updates do not drift too far from prior behavior.
- How deep it goes
- Explains frozen-copy behavior, implicit reward, beta thresholds, and memory tradeoffs without a full mathematical derivation.
- Examples
- Uses chosen and rejected response pairs and a 4B training run to illustrate policy comparison and memory overhead.
- What you can do after
- Apply it via TRL by choosing a strong SFT checkpoint, starting with beta 0.1, and planning for reference-model memory.
Read how LLM Customization and Fine-Tuning explains reference model →
Reinforcement Learning from Human Feedback
A reference model is a frozen copy of a policy that provides a baseline distribution for measuring or penalizing policy deviation.
- How it is used
- Uses a frozen policy baseline to constrain RLHF optimization and compare likelihoods in preference optimization.
- How deep it goes
- Defines the KL-regularized objective and penalty trade-off, and distinguishes PPO’s training-start reference from its batch-time old policy.
- Examples
- Uses a PPO per-token reward penalty and a DPO objective comparing preferred and rejected completion likelihood ratios.
- What you can do after
- Enables readers to evaluate KL penalty strength and distinguish PPO’s reference model from its old policy.
Read how Reinforcement Learning from Human Feedback explains reference model →
What to know first
- Group Relative Policy OptimizationGroup Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled actions or outputs.
- KL regularizationKL regularization is a penalty or constraint based on Kullback–Leibler divergence that limits how far a learned distribution can deviate from a reference distribution.
- reward modelA reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
- reward signalA reward signal is a value or feedback indicating the desirability, benefit, or cost associated with an action, outcome, or behavior.
Related concepts
- RLHFused by · Reinforcement learning from human feedback is a model-training approach that uses human evaluations or preferences as feedback to optimize model behavior.
- KL regularizationused by · KL regularization is a penalty or constraint based on Kullback–Leibler divergence that limits how far a learned distribution can deviate from a reference distribution.
- Direct Preference Optimizationused by · Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.
- low-rank adaptationproduced by · Low-rank adaptation (LoRA) is a parameter-efficient fine-tuning technique that adapts a model by learning low-rank updates to its weights, typically while keeping most original weights fixed.
- policy gradient methodused by · A policy gradient method is a reinforcement-learning method that optimizes a parameterized policy by estimating the gradient of expected return with respect to its parameters.
- Proximal Policy Optimizationpart of · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
Questions about reference model
What is reference model?
A reference model is a fixed policy model whose output distribution provides a baseline for comparing or constraining an optimized policy.
Which Manning books explain reference model?
Applied Reinforcement Learning, Build a DeepSeek Model (From Scratch), LLM Customization and Fine-Tuning and Reinforcement Learning from Human Feedback.
What should I know before learning reference model?
Manning books that teach reference model usually introduce Group Relative Policy Optimization, KL regularization, reward model and reward signal first.
What is reference model part of?
Reference model is part of Proximal Policy Optimization. Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
What is reference model also called?
Reference model is also called reference policy.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



