Chapter 8: Preference optimization: Teaching your model to judge
This introduction explains why preference optimization follows supervised fine-tuning when a model needs to distinguish between acceptable responses of different quality.
After this chapter you will be able to
- SFT and preference distinction. Explain why supervised fine-tuning (SFT) can improve instruction following and formatting yet struggle to distinguish two factually correct responses when one is more helpful, better structured, or more appropriately detailed. Describe how preference optimization addresses that gap using better–worse response pairs.
- SFT quality ceiling. Explain why correct examples can teach a model to produce acceptable responses without teaching it to distinguish good responses from better ones.
- Preference-sensitive quality dimensions. Describe how preferences can refine style consistency, the right amount of detail, and domain-specific safety boundaries beyond what SFT examples establish.
- Response-pair training signal. Trace how preference optimization uses a chosen and a rejected response to the same prompt to shift the model’s own generations.
- Preference pairs and objective. Explain what a DPO training example contains and how DPO uses it to favor one response over another.
- Policy and reference roles. Distinguish the policy model that DPO updates from the frozen reference checkpoint that anchors the update.
- DPO versus RLHF complexity. Explain why DPO avoids the separate reward-model and RL-loop stages used in RLHF.
- Human-graded preference signal. Explain how the IT support dataset turns community judgments into preference pairs: for the same question, a higher-scoring answer is chosen and a lower-scoring answer is rejected. Describe how the contrast can favor an answer that addresses the question directly or follows an organization’s preferred behavior, rather than one that is merely fluent or technically correct.
- TRL preference record format. Describe the three parts of a TRL preference record and explain why they are stored as chat-template message lists rather than raw text strings.
- Preference Pair Source Trade-offs. Distinguish model-model, human-model, log-derived, and judge-model preference pairs by how they are created, what each is useful for, and its main limitation.
- Rejected Response Failure Diversity. Explain why preference data needs varied rejected-response failure modes as well as general data quality and example diversity.
- Conservative DPO training settings. Explain why this DPO run uses a low learning rate, a balanced beta, and one epoch to refine an already-capable SFT model without excessive drift or overfitting.
- Preference training metric interpretation. Explain what reward accuracy and reward margin measure, why held-out results matter more than training accuracy, and what trends indicate a healthy DPO run.
- Reward accuracy and margin. Explain what reward accuracy and reward margin measure, and interpret the training and held-out values in the reference run.
- Metric based next steps. State what to do when DPO metrics fall in the reference ranges versus when reward accuracy is at chance or reward margin stays near zero.
- Preference Pair Training. Trace how DPO uses a chosen and a rejected response to the same prompt: it makes the chosen response more likely and the rejected response less likely.
- Reference Model Drift Constraint. Explain why DPO retains a frozen copy of the pre-DPO checkpoint and how TRL’s beta changes the strength of the constraint.
- Implicit Reward Versus RLHF. Distinguish DPO’s implicit reward from RLHF’s separately trained reward model and PPO step.
- Reproducible DPO pipeline. Describe how the five-step hands-on pipeline runs from data generation through safety testing, and how the table and shell script support running or inspecting it. State the preference-pair split and basic training setup used in this example.
- Quality and safety findings. Distinguish what the reported objective metric and safety check show from the subjective benefit attributed to DPO. Explain why the section does not credit DPO with the earlier format-adherence gain.
- DPO correctness comparison. Explain what the three-way evaluation found about DPO versus SFT on factual IT-support correctness, distinguishing that result from whether DPO learned the preference signal.
- Token-F1 blind spot. Explain why similar Token-F1 scores do not rule out a DPO improvement in conciseness or presentation.
- Technique and data investment. Distinguish when to invest in better SFT data versus preference data, based on whether the workload chiefly needs factual correctness or choices among acceptable answers.
- Single-axis conciseness preferences. Explain how the demonstration isolates conciseness in its preference pairs: the chosen and rejected responses contain the same correct facts, but the chosen response uses fewer words and less filler.
- Conciseness gains and measurement. Describe what the second DPO run improved and why its evaluation used answer length and blinded preference judgments rather than Token-F1. Contrast that result with the lack of measurable improvement in objective correctness from the first run.
- Predefined DPO shipping decision. Explain why go/no-go criteria should be set before DPO training and why the cost of maintaining the DPO stage belongs in the decision.
- Performance gates matched to goals. Distinguish the overall-correctness gate from a gate for the subjective quality DPO is meant to improve, and explain how the chapter's example fared on each.
- Safety and expert review gates. Describe the safety and qualitative checks required before shipping a DPO model.
- Verboseness and style drift. Explain how preference data favoring longer answers can lead to verboseness drift, and describe the response-length check used to detect it.
- Refusal boundary drift. Describe the two directions refusal behavior can drift after preference optimization and how to check for them.
- Mode collapse and diversity. Explain the training conditions associated with mode collapse and describe the response-diversity checks proposed for detecting it.
- DPO reference model overhead. Explain why standard DPO uses more memory and compute than training a single model, and when that overhead may become a bottleneck.
- ORPO single-step alternative. Describe how ORPO combines supervised fine-tuning and preference optimization, and why a team might choose it over a two-stage SFT-then-DPO pipeline.
- SimPO length-aware reward. Explain what SimPO uses instead of a reference model and why it may be relevant when DPO outputs become increasingly verbose.
- Critic-Free GRPO Baseline. Explain why GRPO uses less memory than PPO: it estimates a baseline from sampled responses instead of maintaining a separate critic network.
- Two-Response DPO Equivalence. Describe the condition under which GRPO’s update has the same form as DPO’s loss, and explain the practical implication for tasks already suited to DPO.
- GRPO Advantage Conditions. Explain when GRPO has an advantage over learning from fixed preference pairs: when rewards are programmatically verifiable, exploration matters, or annotated preference data is expensive.
- Group-relative policy update. Trace one GRPO training step: sample a prompt, generate several responses, score them, normalize their rewards within the group, and update the policy toward above-average responses and away from below-average ones, with a KL penalty against the reference policy.
- Group statistics instead of critic. Explain why GRPO uses the group’s own reward statistics as its baseline rather than learning an absolute value estimate with a PPO-style critic.
- GRPO and DPO task fit. Distinguish tasks suited to GRPO from those better suited to DPO: GRPO needs reliably checkable outcomes, a well-defined reward function, and enough compute to generate multiple responses per prompt; DPO is more practical without verifiable rewards.
- GRPO implementation requirements. Describe the changes needed to add GRPO to a DPO pipeline: provide a callable reward function for individual responses, set the group size, and allow GPU memory for generating a group of responses per prompt.
- GRPO compute and reward trade-offs. Explain how generation compute scales with group size and why a cheap, verifiable reward is a favorable fit. Describe the memory advantage the section attributes to removing PPO's critic network.
- RLHF alignment pipeline. Trace the three stages described here: collect human preference annotations, train a reward model on them, and use PPO to optimize the language model against that reward model while staying close to a reference policy.
- RLHF effectiveness and practicality. Explain why RLHF serves as a reference point for DPO, GRPO, and RFT despite being out of reach for most enterprise teams.
- Human preference data collection. Describe how annotators create preference data by choosing the better of two model responses, and explain why reliable labels require substantial annotation effort.
- Reward model training risk. Explain what the separate reward model learns from preference annotations and what can go wrong if it fails to generalize.
- PPO policy optimization demands. Describe how PPO updates the policy using the reward model and why this stage is technically and computationally demanding.
- Enterprise RLHF tradeoff. Explain why many enterprise teams treat RLHF as a fallback, comparing its costs and project timeline with DPO on the same task.
- Novel prompt preference generalization. Explain why a need to learn preferences across new prompt distributions is the stronger reason to consider RLHF, and identify its cost.
- Enterprise RLHF cost trade-off. Describe why most enterprise applications do not need RLHF's marginal gains despite the cases that justify it.
- Verifiable rewards versus preferences. Explain why a task with a verifiable answer can use an automated reward instead of human preference data or a trained reward model.
- Programmable grader optimization loop. Describe the high-level RFT process: a programmable grader supplies rewards, sampled outputs are scored, and the model is updated to favor higher-scoring outputs.
- Verifiable enterprise reward signals. Explain why RFT is practical for enterprise tasks whose outcomes a script can grade, and give examples of outputs paired with their reward checks.
- AMIE diagnostic self-play example. Describe how AMIE combined supervised training with simulated consultations and automated feedback, and explain what its example shows about grading a multi-step process.
- Code-Based Verifiable Rewards. Explain how an RFT reward function scores an output against a checkable answer, using the invoice-extraction example or another example from the section.
- RFT Training Loop Distinction. Trace the RFT training loop and distinguish its rule-based reward signal from an approximate reward-model judgment.
- Minimal Data and Verifier Rewards. Explain why RLVR can improve performance with very little labeled data: a verifier supplies reward feedback by evaluating many model outputs, rather than requiring a large set of labeled examples.
- Indirect Incentive for Reasoning. Explain how RLVR can encourage more structured reasoning even when its reward function checks only the final answer.
- Verifiable reward hacking. Explain how a model can earn a verifiable reward without solving the intended problem: code can hard-code outputs for checked inputs, while math answers can be guesses dressed up with plausible reasoning.
- Reward hacking mitigations. Describe how diverse test cases, partial credit for intermediate reasoning, and held-out evaluation address the weaknesses of checks that look only at a final result.
- RFT compute cost. Explain why RFT costs substantially more compute than a standard fine-tune or DPO run, and distinguish the section’s planning estimates from measured costs.
- RFT applicability criteria. Describe the conditions under which RFT may justify its cost and distinguish tasks with naturally verifiable outcomes from tasks that need another approach.
- Combined Training and Reasoning. Explain that DeepSeek-R1 illustrates how SFT and RL, combined at scale, produced reasoning capabilities that neither step delivered on its own, including multi-step reasoning, self-correction, and longer planning horizons.
- Conditions for Using Cold Start. State the conditions under which cold-start SFT plus RL is relevant: a genuinely multi-step reasoning task, large-scale verifiable rewards, sufficient compute budget, and domain expertise to design SFT examples and reward functions.
- Complementary reasoning training stages. Explain what light SFT and GRPO each contribute to DeepSeek-R1’s two-stage training pattern, and why neither stage works well alone.
- Emergent Reasoning From Rewards. Explain which reasoning behaviors emerged in DeepSeek-R1 and why reward-driven training could produce them even though they were absent from Stage 1 SFT data.
- Enterprise Fit And Narrow Adaptation. Distinguish tasks that benefit from emergent reasoning from routine enterprise tasks, and describe the narrower SFT-plus-GRPO approach suggested for high-value cases.
- Data gate and DPO default. Explain why teams should check SFT data quality before trying preference optimization, then start with DPO for quality or style refinement and move beyond it only if it misses the quality target.
- Verifiable reward methods. Describe when the framework favors GRPO or RFT over preference pairs, and state the compute trade-off.
- Data Quality Before DPO. Explain why an enterprise should inspect its SFT data before investing in preference optimization, and describe the status of the 95/4/1 split.
- Verifiable And Frontier Tasks. Distinguish the tasks suited to GRPO or RFT from those that might justify frontier SFT plus RL, including why the latter is uncommon.
- Length and verboseness bias. Explain why DPO can favor longer responses even when conciseness is preferred, and describe how to check for and address that bias.
- Likelihood displacement and refusals. Explain how likelihood displacement can affect preferred responses and safety refusals, and describe the safety check recommended after DPO training.
- SFT Foundation and Reference Model. Explain why DPO should start from a solid SFT checkpoint and why the frozen pre-DPO reference model matters.
- Limits of DPO for Factuality. Distinguish ordinary preference optimization from fixing factual errors, including the stated exception for factuality-focused pairs.
- Helpfulness Preference Safety Risk. Explain why optimizing DPO for helpfulness can weaken refusal behavior and why safety testing must be repeated after DPO training.
- Regression Versus Baseline Weakness. Distinguish a safety regression introduced by DPO from a failure already present in the SFT model, and describe the release consequence of a lost refusal.
- Refusal Data Repair Strategy. Describe how to address missing refusal behavior and explain why adding refusals to SFT data is more reliable than relying on a small number of DPO safety pairs alone.
- Separate Quality and Safety Preferences. Explain how dual-objective optimization separates task-quality preferences from safety preferences, and why that separation matters in safety-critical applications.
- Safety Regression Testing Workflow. Describe the section’s safety-testing workflow across adaptation steps and state how a detected regression affects deployment.
What it builds on
- The two-stage pattern (chapter 6) · The progression from SFT to preference optimization.
- Hands-on: SFT training workflow (chapter 6) · The SFT training workflow that produces a checkpoint.
- Preparing your SFT checkpoint for preference optimization (chapter 6) · Preparing an SFT checkpoint for preference optimization.
- Why safety does not transfer (chapter 7) · The finding that safety does not transfer through distillation.
- LoRA core idea (chapter 5) · The idea of training a LoRA adapter instead of all model weights.
- Inference (chapter 6) · The SFT inference pattern used as the comparison.
- Evaluation strategy (chapter 6) · The use of held-out evaluation to compare model adaptations.
- Making decisions from evaluation results (chapter 6) · Interpreting evaluation results rather than assuming every adaptation improves every measure.
Section by section
Introduction
This introduction explains why preference optimization follows supervised fine-tuning when a model needs to distinguish between acceptable responses of different quality.
- SFT and preference distinction: Explain why supervised fine-tuning (SFT) can improve instruction following and formatting yet struggle to distinguish two factually correct responses when one is more helpful, better structured, or more appropriately detailed. Describe how preference optimization addresses that gap using better–worse response pairs.
Why it matters: It explains why preference tuning may help even after SFT has improved a model’s output. - Progressive adaptation starting point: Describe how this chapter’s preference optimization builds on an SFT checkpoint, while noting that LoRA and distilled-student models are also valid starting points for DPO.
Why it matters: It clarifies what model the chapter builds on without treating SFT as the only route to DPO.
Key terms: SFT, preference optimization, response pairs, SFT checkpoint, DPO, progressive model adaptation pipeline
8.1 When fine-tuning plateaus
This section explains why SFT can plateau on response quality, how preference optimization uses better–worse response pairs to refine it, and what to watch for when combining it with SFT or distillation.
- SFT quality ceiling: Explain why correct examples can teach a model to produce acceptable responses without teaching it to distinguish good responses from better ones.
Why it matters: It explains why adding more correct examples may not fix a model’s response-quality problems. - Preference-sensitive quality dimensions: Describe how preferences can refine style consistency, the right amount of detail, and domain-specific safety boundaries beyond what SFT examples establish.
Why it matters: These are concrete quality gaps that preference data can target. - Response-pair training signal: Trace how preference optimization uses a chosen and a rejected response to the same prompt to shift the model’s own generations.
Why it matters: It distinguishes preference training from both ordinary SFT and using an LLM to grade answers. - SFT checkpoint and forgetting: Explain the SFT-then-preference-optimization order and why forgetting during SFT can affect the checkpoint selected for DPO.
Why it matters: A preference-tuning stage cannot be planned without considering the quality of the checkpoint it starts from. - Distilled student alignment check: Explain why a distilled student intended for deployment needs its own preference and safety validation.
Why it matters: A well-aligned teacher is not evidence that the deployed student behaves the same way.
Key terms: SFT, SFT quality ceiling, preference optimization, style consistency, helpfulness calibration, safety boundary tuning, chosen, rejected, response pairs, loss function
8.2 Direct preference optimization (DPO): Deep dive
This section explains how DPO learns from preferred responses after SFT and why it is simpler than RLHF.
- Preference pairs and objective: Explain what a DPO training example contains and how DPO uses it to favor one response over another.
Why it matters: This is the training signal that distinguishes DPO from fine-tuning on desired answers alone. - Policy and reference roles: Distinguish the policy model that DPO updates from the frozen reference checkpoint that anchors the update.
Why it matters: The distinction explains how DPO can change preferences without letting the update drift unchecked. - DPO versus RLHF complexity: Explain why DPO avoids the separate reward-model and RL-loop stages used in RLHF.
Why it matters: Removing the reward-model and RL stages is why preference optimization is more practical for many enterprise teams.
Key terms: DPO, preference pairs, chosen, rejected, post-SFT, policy model, frozen reference checkpoint, beta, drift, full-parameter
8.2.1 Building preference data
This section explains how to build DPO preference pairs from human-graded answers and format them for training.
- Human-graded preference signal: Explain how the IT support dataset turns community judgments into preference pairs: for the same question, a higher-scoring answer is chosen and a lower-scoring answer is rejected. Describe how the contrast can favor an answer that addresses the question directly or follows an organization’s preferred behavior, rather than one that is merely fluent or technically correct.
Why it matters: The quality of the chosen-versus-rejected contrast determines what preference the model can learn. - Model-generated pair limitation: Distinguish human-graded pairs from the cheaper shortcut that chooses an SFT model’s answer over a weaker base model’s answer. Explain why this shortcut may help with style or verboseness but does not itself provide human judgment.
Why it matters: The source of a preference pair limits what its chosen label can be taken to mean. - TRL preference record format: Describe the three parts of a TRL preference record and explain why they are stored as chat-template message lists rather than raw text strings.
Why it matters: The record structure lets the trainer use the intended comparison without tying the data to one model’s raw prompt format.
Key terms: preference pairs, chosen, rejected, human-graded preference, human-graded pairs, model-vs-model shortcut, SFT model, weaker base model, TRL convention, prompt
8.2.2 Preference data sources: A taxonomy
This section compares four sources of preference pairs, explains their quality and cost trade-offs, and shows how to choose and diversify data for DPO.
- Preference Pair Source Trade-offs: Distinguish model-model, human-model, log-derived, and judge-model preference pairs by how they are created, what each is useful for, and its main limitation.
Why it matters: The source of a pair affects both what preference the model can learn and what it costs to collect the data. - Judge Bias Safeguards: Explain how bias in a judge-model's rankings can teach the DPO model the wrong preference, and describe the safeguards given here.
Why it matters: Scalable judge-generated data is useful only if the judge rewards the qualities you actually want. - Rejected Response Failure Diversity: Explain why preference data needs varied rejected-response failure modes as well as general data quality and example diversity.
Why it matters: Varied rejected answers keep preference training from reducing quality to a single repeated contrast. - Starting Source and Escalation: Describe the recommended starting source for most DPO teams and when to move to judge-model or human-model pairs.
Why it matters: Starting cheaply lets a team find out whether more nuanced, costly preference data is necessary.
Key terms: model-model pairs, human-model pairs, log-derived pairs, judge-model pairs, chosen, rejected, judge biases, reward hacking, scoring rubric, human judgment
8.2.3 Training with DPOTrainer
This section explains how to train an SFT model on preference pairs with DPO, why the settings are conservative, and how to interpret the resulting metrics and hardware requirements.
- Conservative DPO training settings: Explain why this DPO run uses a low learning rate, a balanced beta, and one epoch to refine an already-capable SFT model without excessive drift or overfitting.
Why it matters: These settings help improve preferences without undoing the capabilities gained through SFT. - Pairwise training and memory: Describe how DPOTrainer differs from SFT training in its inputs, why full-parameter DPO needs more memory, and how the LoRA-DPO variant makes single-GPU training feasible.
Why it matters: The reference model explains both the memory cost of the full run and the appeal of the single-GPU adapter option. - Preference training metric interpretation: Explain what reward accuracy and reward margin measure, why held-out results matter more than training accuracy, and what trends indicate a healthy DPO run.
Why it matters: Reading held-out pairwise metrics together helps distinguish modest real learning from an impressive-looking but misleading training result.
Key terms: learning rate, beta, KL penalty, frozen reference checkpoint, epochs, overfitting, DPOTrainer, preference pairs, policy, frozen reference
8.2.4 Interpreting DPO training metrics
This section explains how to interpret DPO reward accuracy, reward margin, and loss, then use those metrics to decide whether to evaluate the model or repair the preference pairs.
- Reward accuracy and margin: Explain what reward accuracy and reward margin measure, and interpret the training and held-out values in the reference run.
Why it matters: These two metrics show whether the model distinguishes preferred responses and how strongly it does so. - DPO loss comparisons: Describe why DPO loss cannot be compared directly with SFT loss and when comparisons between DPO runs are meaningful.
Why it matters: Comparing loss without its training context can make a DPO run look better or worse for the wrong reason. - Metric based next steps: State what to do when DPO metrics fall in the reference ranges versus when reward accuracy is at chance or reward margin stays near zero.
Why it matters: This check directs effort toward evaluation when the signal is present and toward data repair when it is not.
Key terms: reward accuracy, reward margin, implicit reward, chosen response, rejected response, held-out, DPO loss, SFT loss, log-sigmoid, KL penalty
8.2.5 How DPO works
This section explains how DPO learns from preferred and rejected responses while limiting drift from a frozen reference model, and how that differs from RLHF.
- Preference Pair Training: Trace how DPO uses a chosen and a rejected response to the same prompt: it makes the chosen response more likely and the rejected response less likely.
Why it matters: This is the basic learning signal behind the DPO update. - Reference Model Drift Constraint: Explain why DPO retains a frozen copy of the pre-DPO checkpoint and how TRL’s beta changes the strength of the constraint.
Why it matters: Preference gains need to be balanced against changes to the model’s existing behavior. - Implicit Reward Versus RLHF: Distinguish DPO’s implicit reward from RLHF’s separately trained reward model and PPO step.
Why it matters: It explains why DPO can optimize preferences without RLHF’s separate reward-model training step.
Key terms: DPO, chosen, rejected, policy model, frozen reference model, pre-DPO checkpoint, KL constraint, beta, TRL, implicit reward
8.2.6 End-to-end pipeline summary
Summarizes the runnable DPO pipeline, its training setup, and what its initial evaluation and sample response show.
- Reproducible DPO pipeline: Describe how the five-step hands-on pipeline runs from data generation through safety testing, and how the table and shell script support running or inspecting it. State the preference-pair split and basic training setup used in this example.
Why it matters: This connects the preference data, training run, and checks into a workflow you can reproduce or inspect step by step. - Quality and safety findings: Distinguish what the reported objective metric and safety check show from the subjective benefit attributed to DPO. Explain why the section does not credit DPO with the earlier format-adherence gain.
Why it matters: These findings prevent an unchanged objective score from being mistaken for either a DPO improvement or proof that preference tuning had no value. - Checkpoint inference and response style: Explain how the DPO model is used for inference and describe the response-style difference the section draws between it and the base model.
Why it matters: It clarifies both what artifact the pipeline produces and the practical style difference the example illustrates.
Key terms: DPO hands-on pipeline, human-graded pairs, contamination guard, validation, DPO training, Token-F1, format adherence, conciseness, safety regression, SFT
8.2.7 Evaluating DPO: Three-way comparison
This section compares base, SFT, and DPO models on shared IT-support questions, then explains why flat correctness scores do not capture the subjective qualities DPO is meant to improve.
- Comparable three-model evaluation: Describe how the base, SFT, and DPO models are compared using the same untouched questions, prompts, reference answers, Token-F1 metric, and category splits. Explain why small score differences on this 50-question set should be treated cautiously.
Why it matters: The comparison is useful only if you keep both its shared setup and its limited precision in mind. - DPO correctness comparison: Explain what the three-way evaluation found about DPO versus SFT on factual IT-support correctness, distinguishing that result from whether DPO learned the preference signal.
Why it matters: Learning preferences is not the same as improving factual answers. - Token-F1 blind spot: Explain why similar Token-F1 scores do not rule out a DPO improvement in conciseness or presentation.
Why it matters: A metric can miss a real improvement when it does not measure the quality being optimized. - Technique and data investment: Distinguish when to invest in better SFT data versus preference data, based on whether the workload chiefly needs factual correctness or choices among acceptable answers.
Why it matters: Matching the training method to the desired improvement avoids spending on preference data for a problem it is unlikely to solve.
Key terms: held-out test set, Token-F1, per-category splits, objective correctness, preference signal, SFT, DPO, lexical overlap, conciseness, SFT data
8.2.8 Where DPO wins: A conciseness demonstration
A conciseness experiment shows that DPO can improve a narrowly defined preference between equally correct answers, even when an objective metric misses the gain.
- Single-axis conciseness preferences: Explain how the demonstration isolates conciseness in its preference pairs: the chosen and rejected responses contain the same correct facts, but the chosen response uses fewer words and less filler.
Why it matters: Holding correctness constant makes it possible to teach a preference for tighter answers. - Conciseness gains and measurement: Describe what the second DPO run improved and why its evaluation used answer length and blinded preference judgments rather than Token-F1. Contrast that result with the lack of measurable improvement in objective correctness from the first run.
Why it matters: An evaluation metric must capture the quality the preference data is intended to improve.
Key terms: chosen, rejected, preference dataset, conciseness, Token-F1, blinded judge, objective correctness
8.2.9 Go/no-go criteria for shipping a DPO model
This section gives a precommitted checklist for deciding whether a DPO model is worth shipping, including performance, safety, expert review, and maintenance cost.
- Predefined DPO shipping decision: Explain why go/no-go criteria should be set before DPO training and why the cost of maintaining the DPO stage belongs in the decision.
Why it matters: A decision made against pre-set criteria is less likely to overlook regressions or ongoing work. - Performance gates matched to goals: Distinguish the overall-correctness gate from a gate for the subjective quality DPO is meant to improve, and explain how the chapter's example fared on each.
Why it matters: The release test needs to measure the benefit you sought without mistaking that benefit for an overall accuracy gain. - Safety and expert review gates: Describe the safety and qualitative checks required before shipping a DPO model.
Why it matters: A measured improvement is not a reason to ship a model that regresses on safety or fails on individual production-like requests.
Key terms: go/no-go criteria, DPO layer, SFT checkpoint, Token-F1, SFT baseline, primary subjective dimension, targeted measurement, safety regression, qualitative spot-check, representative production prompts
8.2.10 Monitoring preference-tuned models in production
This section identifies three production drift patterns in DPO models and gives checks for detecting each one.
- Verboseness and style drift: Explain how preference data favoring longer answers can lead to verboseness drift, and describe the response-length check used to detect it.
Why it matters: A model can become less useful through changing answer style without showing an accuracy regression. - Refusal boundary drift: Describe the two directions refusal behavior can drift after preference optimization and how to check for them.
Why it matters: Checking both sides of refusal behavior helps catch safety regressions and unnecessary refusals. - Mode collapse and diversity: Explain the training conditions associated with mode collapse and describe the response-diversity checks proposed for detecting it.
Why it matters: Response-diversity checks can reveal repetitive output that other performance measures may overlook.
Key terms: verboseness drift, DPO evaluation baseline, preference-tuned models, refusal drift, refusal boundaries, safety regression suite, mode collapse, beta, DPO training epochs, unique n-gram ratio
8.2.11 DPO variants: When to consider alternatives
This section explains the costs of DPO’s reference model and when reference-free ORPO or SimPO may be useful alternatives.
- DPO reference model overhead: Explain why standard DPO uses more memory and compute than training a single model, and when that overhead may become a bottleneck.
Why it matters: The reference model’s cost helps you decide whether standard DPO fits your hardware and experiment plans. - ORPO single-step alternative: Describe how ORPO combines supervised fine-tuning and preference optimization, and why a team might choose it over a two-stage SFT-then-DPO pipeline.
Why it matters: ORPO offers a way to avoid both the reference model and a separate SFT-then-DPO sequence. - SimPO length-aware reward: Explain what SimPO uses instead of a reference model and why it may be relevant when DPO outputs become increasingly verbose.
Why it matters: SimPO addresses reference-model overhead and offers an option when answer length drifts upward. - Preference data reuse: Describe what can stay the same when moving from DPO to ORPO or SimPO, and state the section’s recommended starting point.
Why it matters: A shared data format makes it practical to try an alternative without rebuilding preference pairs.
Key terms: frozen reference model, policy model, forward passes, GPU memory, ORPO, Odds Ratio Preference Optimization, odds-ratio loss, preference pairs, SimPO, Simple Preference Optimization
8.3 GRPO: Group relative policy optimization
This section explains GRPO’s memory advantage over PPO, its conditional equivalence to DPO, and the circumstances in which GRPO offers a practical advantage.
- Critic-Free GRPO Baseline: Explain why GRPO uses less memory than PPO: it estimates a baseline from sampled responses instead of maintaining a separate critic network.
Why it matters: The missing critic is the reason given here for GRPO’s memory advantage. - Two-Response DPO Equivalence: Describe the condition under which GRPO’s update has the same form as DPO’s loss, and explain the practical implication for tasks already suited to DPO.
Why it matters: The two-response result cautions against expecting GRPO to improve a subjective preference task merely because it is a different training method. - GRPO Advantage Conditions: Explain when GRPO has an advantage over learning from fixed preference pairs: when rewards are programmatically verifiable, exploration matters, or annotated preference data is expensive.
Why it matters: These conditions identify what GRPO can offer when fixed preference pairs are limiting or costly. - SDFT Implicit Reward: Describe the section’s example of a supervised-looking method behaving like reinforcement learning: SDFT optimizes an implicit reward based on the improvement from showing the model an example.
Why it matters: It reinforces the section’s point that different-looking training approaches can optimize closely related objectives.
Key terms: GRPO, PPO, critic, baseline, sampled responses, 2-GRPO, group size, group normalization, DPO loss, subjective preference axis
8.3.1 How GRPO works
This section explains how a GRPO training step uses relative rewards within a group of responses to update a model without training a critic.
- Group-relative policy update: Trace one GRPO training step: sample a prompt, generate several responses, score them, normalize their rewards within the group, and update the policy toward above-average responses and away from below-average ones, with a KL penalty against the reference policy.
Why it matters: This is the training step behind GRPO’s preference for better-scoring responses. - Group statistics instead of critic: Explain why GRPO uses the group’s own reward statistics as its baseline rather than learning an absolute value estimate with a PPO-style critic.
Why it matters: The choice of baseline explains a practical difference between GRPO and critic-based training.
Key terms: group of G responses, reward function, advantage, policy, KL penalty, reference policy, group normalization, baseline, PPO’s critic
8.3.2 When GRPO excels
This section identifies when GRPO is a good fit for enterprise tasks and why DPO is more practical when rewards cannot be verified.
- GRPO and DPO task fit: Distinguish tasks suited to GRPO from those better suited to DPO: GRPO needs reliably checkable outcomes, a well-defined reward function, and enough compute to generate multiple responses per prompt; DPO is more practical without verifiable rewards.
Why it matters: The choice depends on whether you can verify and reward answers reliably, not just on whether you want better outputs. - Emergent reasoning behaviors: Describe the DeepSeek-R1 example: GRPO-based RL training produced self-verification and exploration of alternative solution paths without explicitly instructing those behaviors in the training data.
Why it matters: The example shows what GRPO-based training has achieved on reasoning tasks with verifiable outcomes.
Key terms: GRPO, DPO, reward function, programmatically checkable answers, preference pairs, DeepSeek-R1, emergent behaviors, self-verification, alternative solution paths
8.3.3 GRPO in practice
This subsection explains what it takes to add GRPO to a DPO-style pipeline and when its compute and memory trade-offs make it practical.
- GRPO implementation requirements: Describe the changes needed to add GRPO to a DPO pipeline: provide a callable reward function for individual responses, set the group size, and allow GPU memory for generating a group of responses per prompt.
Why it matters: These are the concrete changes to plan for when extending an existing DPO pipeline. - GRPO compute and reward trade-offs: Explain how generation compute scales with group size and why a cheap, verifiable reward is a favorable fit. Describe the memory advantage the section attributes to removing PPO's critic network.
Why it matters: The reward's evaluation cost and the number of generated responses determine whether GRPO's simpler training loop is worth its compute bill. - Memory-bound alignment alternatives: Distinguish the memory-saving option presented for memory-bound GRPO-style training from the section's production default.
Why it matters: Memory pressure may justify evaluating alternatives without assuming they have displaced the production default.
Key terms: GRPO, DPOTrainer, callable reward function, group size 𝐺, float reward score, verifiable reward functions, critic network, PPO, judge model, ORPO
8.4 RLHF: Why most enterprises skip it
Introduces RLHF’s three-stage alignment pipeline and explains why its effectiveness does not make it practical for most enterprise teams.
- RLHF alignment pipeline: Trace the three stages described here: collect human preference annotations, train a reward model on them, and use PPO to optimize the language model against that reward model while staying close to a reference policy.
Why it matters: This pipeline is the baseline for understanding what the chapter’s alternatives change. - RLHF effectiveness and practicality: Explain why RLHF serves as a reference point for DPO, GRPO, and RFT despite being out of reach for most enterprise teams.
Why it matters: RLHF provides a useful benchmark even when it is not a practical choice for your team.
Key terms: RLHF, human preference annotations, reward model, PPO, reference policy, DPO, GRPO, RFT
8.4.1 The RLHF pipeline
This section traces RLHF’s three-stage pipeline and explains why its data, model, and compute requirements are greater than DPO’s.
- Human preference data collection: Describe how annotators create preference data by choosing the better of two model responses, and explain why reliable labels require substantial annotation effort.
Why it matters: The first stage is already a substantial part of RLHF’s cost and effort. - Reward model training risk: Explain what the separate reward model learns from preference annotations and what can go wrong if it fails to generalize.
Why it matters: RLHF depends on a reward signal that reflects human preferences beyond the examples used to train it. - PPO policy optimization demands: Describe how PPO updates the policy using the reward model and why this stage is technically and computationally demanding.
Why it matters: The optimization stage adds substantial tuning and memory requirements beyond ordinary fine-tuning. - RLHF and DPO resource contrast: Distinguish RLHF’s three-stage, four-model pipeline from DPO’s single supervised step with a policy and frozen reference, including the section’s approximate cost, time, and quality comparison.
Why it matters: The comparison makes the practical resource trade-off between the two approaches clear.
Key terms: preference data collection, annotated pairs, inter-annotator disagreement, reward model, human preference, overfits, policy, PPO, reference, value network
8.4.2 Why enterprises skip it
This section explains why enterprises often treat RLHF as a fallback despite its successes: its cost, staffing needs, training complexity, and reward-hacking risk can outweigh modest quality gains.
- Enterprise RLHF tradeoff: Explain why many enterprise teams treat RLHF as a fallback, comparing its costs and project timeline with DPO on the same task.
Why it matters: The comparison helps you judge whether RLHF’s potential gain warrants an enterprise-scale investment. - PPO training fragility: Describe how PPO hyperparameter sensitivity makes RLHF training and debugging difficult for enterprise teams.
Why it matters: Training fragility adds engineering risk beyond RLHF’s direct costs. - Reward hacking risk: Explain why optimizing an imperfect reward model can produce high-scoring responses that people judge poorly, and describe the operational response noted here.
Why it matters: A rising reward score does not by itself mean users will receive better answers.
Key terms: RLHF, DPO, annotation, compute, PPO, learning rate, KL coefficient, batch size, reward hacking, reward model
8.4.3 When RLHF is unavoidable
This section explains the two situations that can justify RLHF over DPO and why its extra cost is usually unnecessary for enterprise applications.
- Fine-grained preference signals: Explain when graded response judgments favor RLHF over DPO, while noting that rankings can also be converted into DPO pairs.
Why it matters: This distinction helps you decide whether the gradations in your judgments warrant a more expensive approach. - Novel prompt preference generalization: Explain why a need to learn preferences across new prompt distributions is the stronger reason to consider RLHF, and identify its cost.
Why it matters: The potential to handle new kinds of prompts is a stronger justification for RLHF than graded ratings alone. - Enterprise RLHF cost trade-off: Describe why most enterprise applications do not need RLHF's marginal gains despite the cases that justify it.
Why it matters: It keeps the decision focused on whether a possible quality gain is valuable enough to justify RLHF's cost.
Key terms: reward model, ordinal rankings, scalar scores, binary pairs, prompt distributions, online RL, preference learning, DPO, RLHF, quality baseline
8.5 Reinforcement fine-tuning (RFT) with verifiable rewards
Introduces reinforcement fine-tuning as a way to optimize tasks with verifiable answers using automated rewards instead of human preferences.
- Verifiable rewards versus preferences: Explain why a task with a verifiable answer can use an automated reward instead of human preference data or a trained reward model.
Why it matters: This distinction helps you recognize when automated checks can replace costly preference collection. - Programmable grader optimization loop: Describe the high-level RFT process: a programmable grader supplies rewards, sampled outputs are scored, and the model is updated to favor higher-scoring outputs.
Why it matters: This is the basic connection between an automated correctness check and a model that produces better-scoring answers.
Key terms: verifiable answer, Reinforcement Learning with Verifiable Rewards, Reinforcement Fine-Tuning, RFT, automated reward functions, programmable grader, reward, sampled candidates
8.5.1 Enterprise applications of RFT
This section identifies enterprise tasks suited to verifiable rewards and shows how reward-based training can extend from grading single outputs to grading multi-step conversations and workflows.
- Verifiable enterprise reward signals: Explain why RFT is practical for enterprise tasks whose outcomes a script can grade, and give examples of outputs paired with their reward checks.
Why it matters: A checkable outcome tells you whether RFT has a useful reward signal for your task. - AMIE diagnostic self-play example: Describe how AMIE combined supervised training with simulated consultations and automated feedback, and explain what its example shows about grading a multi-step process.
Why it matters: AMIE illustrates how a verifiable reward can cover an extended interaction, not only a short answer. - Whole workflow trajectory grading: Distinguish grading one output from grading an agent's trajectory through a simulated workflow, including the main requirement for the environment.
Why it matters: For tool-using agents, the quality of the simulated workflow matters as much as the grading rule.
Key terms: RFT, reward function, verifiable outcomes, human preference annotations, AMIE, SFT, self-play reinforcement learning, simulated patient agent, automated critic, differential diagnosis
8.5.2 How RFT works
This section explains how RFT uses code-based rewards to score model outputs and update the model toward verifiably correct responses.
- Code-Based Verifiable Rewards: Explain how an RFT reward function scores an output against a checkable answer, using the invoice-extraction example or another example from the section.
Why it matters: The verifier determines what counts as a better response during RFT. - RFT Training Loop Distinction: Trace the RFT training loop and distinguish its rule-based reward signal from an approximate reward-model judgment.
Why it matters: This distinction explains what changes when training uses verifiable rewards.
Key terms: reward function, ground truth, gold standard, training loop, policy, reward signal, reward model
8.5.3 The 2025 RLVR findings
This section explains two 2025 findings about RLVR: it can improve performance with very little labeled data, and final-answer rewards can indirectly encourage structured reasoning.
- Minimal Data and Verifier Rewards: Explain why RLVR can improve performance with very little labeled data: a verifier supplies reward feedback by evaluating many model outputs, rather than requiring a large set of labeled examples.
Why it matters: It explains why reward-driven training may be useful even when labeled examples are scarce. - Indirect Incentive for Reasoning: Explain how RLVR can encourage more structured reasoning even when its reward function checks only the final answer.
Why it matters: It distinguishes what the verifier checks from the reasoning behavior training may foster.
Key terms: RLVR, verifier function, reward signal, MATH-500, structured reasoning chains, reward function, final answer
8.5.4 Key challenge: Reward hacking
The section explains how models can game verifiable rewards and gives three ways to reduce that risk.
- Verifiable reward hacking: Explain how a model can earn a verifiable reward without solving the intended problem: code can hard-code outputs for checked inputs, while math answers can be guesses dressed up with plausible reasoning.
Why it matters: A passing result does not necessarily mean the model solved the problem as intended. - Reward hacking mitigations: Describe how diverse test cases, partial credit for intermediate reasoning, and held-out evaluation address the weaknesses of checks that look only at a final result.
Why it matters: These checks help reveal whether a model has learned to solve problems rather than merely pass a narrow reward test.
Key terms: verifiable rewards, reward hacking, hard-coded outputs, diverse test cases, partial credit, held-out evaluation, generalization
8.5.5 Cost and applicability
This section explains when RFT’s high compute cost is justified and describes DAPO techniques for making group-based RL training more robust and efficient.
- RFT compute cost: Explain why RFT costs substantially more compute than a standard fine-tune or DPO run, and distinguish the section’s planning estimates from measured costs.
Why it matters: The cost difference is large enough to shape whether RFT is a practical choice. - RFT applicability criteria: Describe the conditions under which RFT may justify its cost and distinguish tasks with naturally verifiable outcomes from tasks that need another approach.
Why it matters: A verifiable task alone is not enough to justify an extended, costly RFT run. - DAPO robustness techniques: Explain the group-based RL failures DAPO addresses and describe how its four techniques respond to them.
Why it matters: These techniques offer practical ways to avoid wasted training and biased outputs in group-based RL.
Key terms: RFT, GRPO step, DPO, compute multiplier, naturally verifiable outcomes, quality gap, compute budget, preference pairs, reward model, DAPO
8.6 Cold-start SFT + RL: The DeepSeek-R1 pattern
Introduces DeepSeek-R1 as an illustration of how SFT and RL can combine to produce reasoning capabilities, and identifies when that costly approach is relevant.
- Combined Training and Reasoning: Explain that DeepSeek-R1 illustrates how SFT and RL, combined at scale, produced reasoning capabilities that neither step delivered on its own, including multi-step reasoning, self-correction, and longer planning horizons.
Why it matters: It explains why the pattern is interesting for demanding enterprise tasks without treating it as a routine training choice. - Conditions for Using Cold Start: State the conditions under which cold-start SFT plus RL is relevant: a genuinely multi-step reasoning task, large-scale verifiable rewards, sufficient compute budget, and domain expertise to design SFT examples and reward functions.
Why it matters: These constraints help distinguish a plausible use case from one better served by a less demanding approach.
Key terms: DeepSeek-R1, SFT, RL, emergent reasoning capabilities, cold-start SFT + RL, genuine multi-step reasoning, verifiable rewards, Stage 1 SFT examples, reward functions
8.6.1 The two-stage pattern
Explains how DeepSeek-R1 combines light supervised fine-tuning with reward-driven GRPO so a model learns both the form and the purpose of structured reasoning.
- Complementary reasoning training stages: Explain what light SFT and GRPO each contribute to DeepSeek-R1’s two-stage training pattern, and why neither stage works well alone.
Why it matters: The distinction explains why a convincing-looking reasoning trace is not, by itself, evidence of a correct answer.
Key terms: Light SFT, GRPO, verifiable rewards, chain-of-thought reasoning, explicit verification steps
8.6.2 Emergent behaviors
This section explains how reward-driven training can produce reasoning behaviors not taught in SFT, where those behaviors are useful, and why most enterprise teams should consider cheaper or narrower approaches.
- Emergent Reasoning From Rewards: Explain which reasoning behaviors emerged in DeepSeek-R1 and why reward-driven training could produce them even though they were absent from Stage 1 SFT data.
Why it matters: The distinction explains why a reasoning-trained model may investigate a root cause rather than stop at a surface symptom. - Enterprise Fit And Narrow Adaptation: Distinguish tasks that benefit from emergent reasoning from routine enterprise tasks, and describe the narrower SFT-plus-GRPO approach suggested for high-value cases.
Why it matters: It helps teams reserve expensive reasoning training for problems that need it. - SDFT As Cold-Start Alternative: Describe why the section raises SDFT as a possible cold-start alternative, how its training signal differs from standard SFT, and what limits the recommendation.
Why it matters: It identifies a possible way to reduce forgetting before preference or reinforcement training without treating early research as settled practice.
Key terms: emergent behaviors, self-reflection, alternative paths, error correction, structured verification, GRPO, root cause analysis, DPO, Stage-1 SFT, small-scale GRPO
8.7 Decision framework: Which technique?
The section gives an SFT-data-quality-first framework for choosing among preference optimization techniques based on the task, quality target, and cost.
- Data gate and DPO default: Explain why teams should check SFT data quality before trying preference optimization, then start with DPO for quality or style refinement and move beyond it only if it misses the quality target.
Why it matters: It keeps a team from paying for a more complex method when data improvements or DPO may be enough. - Verifiable reward methods: Describe when the framework favors GRPO or RFT over preference pairs, and state the compute trade-off.
Why it matters: A checkable outcome can justify a richer training signal, but only when its value warrants substantially more compute. - Exceptional optimization choices: Distinguish the limited cases for RLHF and cold-start SFT+RL from the usual DPO starting point.
Why it matters: These exceptions help a team recognize when a demanding objective may justify departing from DPO.
Key terms: SFT-data-quality gate, DPO, quality target, preference pairs, GRPO, RFT, verifiable rewards, RLHF, binary pairs, PPO complexity
8.8 Honest assessment: Failure modes and practical limits
8.8.1 The 95/4/1 breakdown
This section uses an illustrative 95/4/1 split to help enterprises choose between improving SFT data, using DPO for targeted gains, and pursuing more demanding reward-based methods.
- Data Quality Before DPO: Explain why an enterprise should inspect its SFT data before investing in preference optimization, and describe the status of the 95/4/1 split.
Why it matters: Checking the data first can prevent spending on an optimization method that does not address the main quality problem. - Targeted Gains From DPO: Describe when DPO can be a modest-cost way to improve a specific response quality and what the section’s conciseness example demonstrates.
Why it matters: DPO is most useful when the team has a specific response preference to improve, rather than a general hope of better quality. - Verifiable And Frontier Tasks: Distinguish the tasks suited to GRPO or RFT from those that might justify frontier SFT plus RL, including why the latter is uncommon.
Why it matters: The kind of feedback a task provides—and the resources available—help determine whether a more demanding method is warranted.
Key terms: 95/4/1 split, SFT data, target distribution, DPO, preference pairs, conciseness, SFT model, GRPO, RFT, verifiable domains
8.8.2 Known DPO failure modes
This section explains two DPO failure modes that can undermine enterprise deployments and how to detect or mitigate them.
- Length and verboseness bias: Explain why DPO can favor longer responses even when conciseness is preferred, and describe how to check for and address that bias.
Why it matters: A preference-tuned model can appear to improve while becoming too wordy for its intended use. - Likelihood displacement and refusals: Explain how likelihood displacement can affect preferred responses and safety refusals, and describe the safety check recommended after DPO training.
Why it matters: Overall quality scores can miss a safety regression in a preference-tuned model.
Key terms: DPO, length and verboseness bias, output length distributions, length-controlled evaluation, SimPO, length-desensitization techniques, likelihood displacement, safety-related preference pairs, chosen response, refusal rates
8.8.3 Common mistakes
This section identifies common enterprise DPO mistakes involving the SFT foundation, preference-pair quality, the reference model, and factual accuracy.
- SFT Foundation and Reference Model: Explain why DPO should start from a solid SFT checkpoint and why the frozen pre-DPO reference model matters.
Why it matters: A DPO run cannot compensate for a poor starting checkpoint. - Preference Pair Quantity and Quality: Describe the production guidance for preference-pair counts and explain why pair quality matters more than quantity.
Why it matters: Collecting more comparisons helps only if their quality is high enough to support preference learning. - Limits of DPO for Factuality: Distinguish ordinary preference optimization from fixing factual errors, including the stated exception for factuality-focused pairs.
Why it matters: Choosing DPO to fix factual errors without targeted data and evaluation can leave hallucinations unresolved.
Key terms: SFT, reference model, frozen pre-DPO checkpoint, preference pairs, pair quality, factual accuracy, hallucinations, retrieval augmentation, factuality-focused preference data
8.9 Preference optimization and safety
This section explains how DPO can affect safety, how to distinguish a new safety regression from an existing SFT weakness, and how to respond before release.
- Helpfulness Preference Safety Risk: Explain why optimizing DPO for helpfulness can weaken refusal behavior and why safety testing must be repeated after DPO training.
Why it matters: A model can become more responsive while becoming less willing to refuse harmful requests. - Regression Versus Baseline Weakness: Distinguish a safety regression introduced by DPO from a failure already present in the SFT model, and describe the release consequence of a lost refusal.
Why it matters: A comparison can show that DPO did not make safety worse without showing that the model is safe to ship. - Refusal Data Repair Strategy: Describe how to address missing refusal behavior and explain why adding refusals to SFT data is more reliable than relying on a small number of DPO safety pairs alone.
Why it matters: Detecting an unsafe baseline is not enough; the training data must support the refusal behavior you want to retain.
Key terms: DPO, helpfulness, refusal, likelihood displacement, safety regression testing, safety regression, SFT model, DPO model, release gate, refusal rate
8.9.1 Dual-objective optimization
This section explains how to keep preference optimization from weakening safety and how to test for safety regressions after model adaptation.
- Separate Quality and Safety Preferences: Explain how dual-objective optimization separates task-quality preferences from safety preferences, and why that separation matters in safety-critical applications.
Why it matters: In regulated applications, a safety regression can be a compliance violation, not merely a drop in quality. - Safety Regression Testing Workflow: Describe the section’s safety-testing workflow across adaptation steps and state how a detected regression affects deployment.
Why it matters: A gain in helpfulness does not justify shipping a model that fails a safety test it previously passed.
Key terms: dual-objective optimization, task quality, safety, dual-objective DPO, safety test suite, harmful request refusal, uncertainty acknowledgment, bias neutrality, general knowledge retention, safety regression
8.10 Summary
8.11 References
This section lists sources cited in the chapter without explaining additional concepts or procedures.
Where this is used later
- Gating retrains (chapter 9) · Provides a concrete example of gating a model change before deployment.
- Model drift: Understanding and responding (chapter 9) · Places this preference-specific style check alongside the broader drift detection framework.
- Security and abuse monitoring (chapter 9) · Connects fixed-prompt safety checks to production security and abuse monitoring.
- When to retrain for safety (chapter 9) · Informs decisions about when safety failures require retraining.
- Production prompt monitoring (chapter 9) · Safety boundaries that can shift during preference optimization.
- Running the safety monitor (chapter 9) · Safety monitoring continues the concern that adaptation must preserve safety behavior.
Concepts explained in this chapter
- Direct Preference OptimizationA method that tunes a model directly from preference comparisons without separately training a reward model.
- preference pairA training example containing two responses to the same prompt and a label indicating which is preferred.
- supervised fine-tuningFurther training a pretrained model on labeled input-output examples to adapt its behavior.
- reference modelA fixed policy distribution that serves as the comparison point for constraining changes to an optimized policy.
- preference tuningTraining a model from preference signals so that it favors desired responses over less desired ones.
- reinforcement learning with verifiable rewardsA method of fine-tuning a model with reinforcement learning using feedback on its outputs.
- KL regularizationA constraint based on Kullback–Leibler divergence that limits how far a policy distribution can move from a reference distribution.
- Group Relative Policy OptimizationA reinforcement-learning method that optimizes a policy using relative comparisons among groups of sampled outputs instead of a separate critic network.
- preference dataA collection of judgments or comparisons expressing preferences between candidate model outputs.
- reward functionA function that assigns a numerical score to an output, action, or sequence of actions to represent its quality.
- preference marginThe difference between the implicit rewards assigned to a chosen response and a rejected response in a preference pair.
- reward accuracyThe proportion of preference pairs for which a model assigns a higher implicit reward to the chosen response than to the rejected response.
- chosen responseThe candidate response in a preference pair that a human or automated system prefers.
- rejected responseThe candidate response in a preference pair that is less preferred than its paired alternative.
- DPOTrainerA TRL trainer for optimizing transformer language models from preference pairs using Direct Preference Optimization.
- regressionA change that makes a model less likely to decline requests outside its competence.
- safety regression suiteThe practice of repeatedly evaluating model changes for newly introduced safety failures.
- RLHFA training approach that uses human preference feedback to shape a model, typically through a reward model and reinforcement-learning optimization.
- reinforcement learningA machine-learning approach in which an agent learns a policy through reward signals from interactions or sampled outcomes.
- self-distillation fine-tuningA fine-tuning method in which a model conditioned on an expert demonstration trains the same model conditioned on the prompt alone.
- refusal behaviorA model's act of declining to fulfill a request, often because complying would be inappropriate or unsafe.
- training datasetThe collection of examples used to fit or adapt a model during training.
- reward modelA model trained to assign scores representing the desirability of outputs, often from human preference judgments.
- spot-checkingHuman review of a sample of examples to identify quality problems that automated checks may miss.
- failure modeA recurring way in which a retrieval-augmented generation system can produce incorrect or degraded behavior in production.
- gradient checkpointingA memory-saving training technique that discards intermediate activations and recomputes them as needed during backpropagation.
- learning rateThe learning rate is a training hyperparameter that controls the size of parameter updates during optimization.
- promptThe input supplied to a model, potentially containing instructions, context, and examples.
- full fine-tuningFine-tuning that updates the model's full set of weights rather than only a subset or adapter.
- baselineA reference performance level against which model changes are evaluated.
- refusal rateRefusal rate is the proportion of tested requests that a model refuses, with the request set determining whether it measures appropriate or harmful-request refusals.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
