liveBook · Manning

Reference model

Also called reference policy

A reference model is a fixed policy model whose output distribution provides a baseline for comparing or constraining an optimized policy.

How 4 Manning books explain it

Applied Reinforcement Learning

A reference model is a fixed model whose policy distribution provides a comparison point during policy training.

How it is used
Constrains policy training by penalizing divergence from a trusted frozen starting policy.
How deep it goes
Explains the KL penalty’s role and distinguishes the reference model from PPO’s data-generating old policy.
Examples
Uses a medical chatbot’s fluency as a behavior to preserve and shows LoRA adapters switched off for a reference pass.
What you can do after
Apply it via LoRA by disabling adapters to compute reference log-probabilities without holding a second full model copy.

Read how Applied Reinforcement Learning explains reference model →

Build a DeepSeek Model (From Scratch)

A reference policy is a fixed or separately maintained policy whose output probabilities provide a comparison distribution during policy optimization.

How it is used
Anchors PPO- and GRPO-style policy optimization against excessive distributional drift.
How deep it goes
Explains freezing, matched token scoring, KL penalties, and implementation caveats, while deferring complete production trainer details.
Examples
Uses a clipped GRPO equation, a PyTorch loss listing, and a training-step skeleton that scores completions under both policies.
What you can do after
Operate/configure a frozen reference policy and include its token-level KL term in a GRPO-style update.

Read how Build a DeepSeek Model (From Scratch) explains reference model →

LLM Customization and Fine-Tuning

A fixed policy distribution that serves as the comparison point for constraining changes to an optimized policy.

How it is used
Anchors preference optimization to a frozen pre-DPO checkpoint so policy updates do not drift too far from prior behavior.
How deep it goes
Explains frozen-copy behavior, implicit reward, beta thresholds, and memory tradeoffs without a full mathematical derivation.
Examples
Uses chosen and rejected response pairs and a 4B training run to illustrate policy comparison and memory overhead.
What you can do after
Apply it via TRL by choosing a strong SFT checkpoint, starting with beta 0.1, and planning for reference-model memory.

Read how LLM Customization and Fine-Tuning explains reference model →

Reinforcement Learning from Human Feedback

A reference model is a frozen copy of a policy that provides a baseline distribution for measuring or penalizing policy deviation.

How it is used
Uses a frozen policy baseline to constrain RLHF optimization and compare likelihoods in preference optimization.
How deep it goes
Defines the KL-regularized objective and penalty trade-off, and distinguishes PPO’s training-start reference from its batch-time old policy.
Examples
Uses a PPO per-token reward penalty and a DPO objective comparing preferred and rejected completion likelihood ratios.
What you can do after
Enables readers to evaluate KL penalty strength and distinguish PPO’s reference model from its old policy.

Read how Reinforcement Learning from Human Feedback explains reference model →

What to know first

Related concepts

Questions about reference model

What is reference model?

A reference model is a fixed policy model whose output distribution provides a baseline for comparing or constraining an optimized policy.

Which Manning books explain reference model?

Applied Reinforcement Learning, Build a DeepSeek Model (From Scratch), LLM Customization and Fine-Tuning and Reinforcement Learning from Human Feedback.

What should I know before learning reference model?

Manning books that teach reference model usually introduce Group Relative Policy Optimization, KL regularization, reward model and reward signal first.

What is reference model part of?

Reference model is part of Proximal Policy Optimization. Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.

What is reference model also called?

Reference model is also called reference policy.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.