liveBook · Manning

Chapter 8: Preference optimization: Teaching your model to judge

What you will learn in chapter 8 of LLM Customization and Fine-Tuning

This introduction explains why preference optimization follows supervised fine-tuning when a model needs to distinguish between acceptable responses of different quality.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction explains why preference optimization follows supervised fine-tuning when a model needs to distinguish between acceptable responses of different quality.

Key terms: SFT, preference optimization, response pairs, SFT checkpoint, DPO, progressive model adaptation pipeline

8.1 When fine-tuning plateaus

This section explains why SFT can plateau on response quality, how preference optimization uses better–worse response pairs to refine it, and what to watch for when combining it with SFT or distillation.

Key terms: SFT, SFT quality ceiling, preference optimization, style consistency, helpfulness calibration, safety boundary tuning, chosen, rejected, response pairs, loss function

8.2 Direct preference optimization (DPO): Deep dive

This section explains how DPO learns from preferred responses after SFT and why it is simpler than RLHF.

Key terms: DPO, preference pairs, chosen, rejected, post-SFT, policy model, frozen reference checkpoint, beta, drift, full-parameter

8.2.1 Building preference data

This section explains how to build DPO preference pairs from human-graded answers and format them for training.

Key terms: preference pairs, chosen, rejected, human-graded preference, human-graded pairs, model-vs-model shortcut, SFT model, weaker base model, TRL convention, prompt

8.2.2 Preference data sources: A taxonomy

This section compares four sources of preference pairs, explains their quality and cost trade-offs, and shows how to choose and diversify data for DPO.

Key terms: model-model pairs, human-model pairs, log-derived pairs, judge-model pairs, chosen, rejected, judge biases, reward hacking, scoring rubric, human judgment

8.2.3 Training with DPOTrainer

This section explains how to train an SFT model on preference pairs with DPO, why the settings are conservative, and how to interpret the resulting metrics and hardware requirements.

Key terms: learning rate, beta, KL penalty, frozen reference checkpoint, epochs, overfitting, DPOTrainer, preference pairs, policy, frozen reference

8.2.4 Interpreting DPO training metrics

This section explains how to interpret DPO reward accuracy, reward margin, and loss, then use those metrics to decide whether to evaluate the model or repair the preference pairs.

Key terms: reward accuracy, reward margin, implicit reward, chosen response, rejected response, held-out, DPO loss, SFT loss, log-sigmoid, KL penalty

8.2.5 How DPO works

This section explains how DPO learns from preferred and rejected responses while limiting drift from a frozen reference model, and how that differs from RLHF.

Key terms: DPO, chosen, rejected, policy model, frozen reference model, pre-DPO checkpoint, KL constraint, beta, TRL, implicit reward

8.2.6 End-to-end pipeline summary

Summarizes the runnable DPO pipeline, its training setup, and what its initial evaluation and sample response show.

Key terms: DPO hands-on pipeline, human-graded pairs, contamination guard, validation, DPO training, Token-F1, format adherence, conciseness, safety regression, SFT

8.2.7 Evaluating DPO: Three-way comparison

This section compares base, SFT, and DPO models on shared IT-support questions, then explains why flat correctness scores do not capture the subjective qualities DPO is meant to improve.

Key terms: held-out test set, Token-F1, per-category splits, objective correctness, preference signal, SFT, DPO, lexical overlap, conciseness, SFT data

8.2.8 Where DPO wins: A conciseness demonstration

A conciseness experiment shows that DPO can improve a narrowly defined preference between equally correct answers, even when an objective metric misses the gain.

Key terms: chosen, rejected, preference dataset, conciseness, Token-F1, blinded judge, objective correctness

8.2.9 Go/no-go criteria for shipping a DPO model

This section gives a precommitted checklist for deciding whether a DPO model is worth shipping, including performance, safety, expert review, and maintenance cost.

Key terms: go/no-go criteria, DPO layer, SFT checkpoint, Token-F1, SFT baseline, primary subjective dimension, targeted measurement, safety regression, qualitative spot-check, representative production prompts

8.2.10 Monitoring preference-tuned models in production

This section identifies three production drift patterns in DPO models and gives checks for detecting each one.

Key terms: verboseness drift, DPO evaluation baseline, preference-tuned models, refusal drift, refusal boundaries, safety regression suite, mode collapse, beta, DPO training epochs, unique n-gram ratio

8.2.11 DPO variants: When to consider alternatives

This section explains the costs of DPO’s reference model and when reference-free ORPO or SimPO may be useful alternatives.

Key terms: frozen reference model, policy model, forward passes, GPU memory, ORPO, Odds Ratio Preference Optimization, odds-ratio loss, preference pairs, SimPO, Simple Preference Optimization

8.3 GRPO: Group relative policy optimization

This section explains GRPO’s memory advantage over PPO, its conditional equivalence to DPO, and the circumstances in which GRPO offers a practical advantage.

Key terms: GRPO, PPO, critic, baseline, sampled responses, 2-GRPO, group size, group normalization, DPO loss, subjective preference axis

8.3.1 How GRPO works

This section explains how a GRPO training step uses relative rewards within a group of responses to update a model without training a critic.

Key terms: group of G responses, reward function, advantage, policy, KL penalty, reference policy, group normalization, baseline, PPO’s critic

8.3.2 When GRPO excels

This section identifies when GRPO is a good fit for enterprise tasks and why DPO is more practical when rewards cannot be verified.

Key terms: GRPO, DPO, reward function, programmatically checkable answers, preference pairs, DeepSeek-R1, emergent behaviors, self-verification, alternative solution paths

8.3.3 GRPO in practice

This subsection explains what it takes to add GRPO to a DPO-style pipeline and when its compute and memory trade-offs make it practical.

Key terms: GRPO, DPOTrainer, callable reward function, group size 𝐺, float reward score, verifiable reward functions, critic network, PPO, judge model, ORPO

8.4 RLHF: Why most enterprises skip it

Introduces RLHF’s three-stage alignment pipeline and explains why its effectiveness does not make it practical for most enterprise teams.

Key terms: RLHF, human preference annotations, reward model, PPO, reference policy, DPO, GRPO, RFT

8.4.1 The RLHF pipeline

This section traces RLHF’s three-stage pipeline and explains why its data, model, and compute requirements are greater than DPO’s.

Key terms: preference data collection, annotated pairs, inter-annotator disagreement, reward model, human preference, overfits, policy, PPO, reference, value network

8.4.2 Why enterprises skip it

This section explains why enterprises often treat RLHF as a fallback despite its successes: its cost, staffing needs, training complexity, and reward-hacking risk can outweigh modest quality gains.

Key terms: RLHF, DPO, annotation, compute, PPO, learning rate, KL coefficient, batch size, reward hacking, reward model

8.4.3 When RLHF is unavoidable

This section explains the two situations that can justify RLHF over DPO and why its extra cost is usually unnecessary for enterprise applications.

Key terms: reward model, ordinal rankings, scalar scores, binary pairs, prompt distributions, online RL, preference learning, DPO, RLHF, quality baseline

8.5 Reinforcement fine-tuning (RFT) with verifiable rewards

Introduces reinforcement fine-tuning as a way to optimize tasks with verifiable answers using automated rewards instead of human preferences.

Key terms: verifiable answer, Reinforcement Learning with Verifiable Rewards, Reinforcement Fine-Tuning, RFT, automated reward functions, programmable grader, reward, sampled candidates

8.5.1 Enterprise applications of RFT

This section identifies enterprise tasks suited to verifiable rewards and shows how reward-based training can extend from grading single outputs to grading multi-step conversations and workflows.

Key terms: RFT, reward function, verifiable outcomes, human preference annotations, AMIE, SFT, self-play reinforcement learning, simulated patient agent, automated critic, differential diagnosis

8.5.2 How RFT works

This section explains how RFT uses code-based rewards to score model outputs and update the model toward verifiably correct responses.

Key terms: reward function, ground truth, gold standard, training loop, policy, reward signal, reward model

8.5.3 The 2025 RLVR findings

This section explains two 2025 findings about RLVR: it can improve performance with very little labeled data, and final-answer rewards can indirectly encourage structured reasoning.

Key terms: RLVR, verifier function, reward signal, MATH-500, structured reasoning chains, reward function, final answer

8.5.4 Key challenge: Reward hacking

The section explains how models can game verifiable rewards and gives three ways to reduce that risk.

Key terms: verifiable rewards, reward hacking, hard-coded outputs, diverse test cases, partial credit, held-out evaluation, generalization

8.5.5 Cost and applicability

This section explains when RFT’s high compute cost is justified and describes DAPO techniques for making group-based RL training more robust and efficient.

Key terms: RFT, GRPO step, DPO, compute multiplier, naturally verifiable outcomes, quality gap, compute budget, preference pairs, reward model, DAPO

8.6 Cold-start SFT + RL: The DeepSeek-R1 pattern

Introduces DeepSeek-R1 as an illustration of how SFT and RL can combine to produce reasoning capabilities, and identifies when that costly approach is relevant.

Key terms: DeepSeek-R1, SFT, RL, emergent reasoning capabilities, cold-start SFT + RL, genuine multi-step reasoning, verifiable rewards, Stage 1 SFT examples, reward functions

8.6.1 The two-stage pattern

Explains how DeepSeek-R1 combines light supervised fine-tuning with reward-driven GRPO so a model learns both the form and the purpose of structured reasoning.

Key terms: Light SFT, GRPO, verifiable rewards, chain-of-thought reasoning, explicit verification steps

8.6.2 Emergent behaviors

This section explains how reward-driven training can produce reasoning behaviors not taught in SFT, where those behaviors are useful, and why most enterprise teams should consider cheaper or narrower approaches.

Key terms: emergent behaviors, self-reflection, alternative paths, error correction, structured verification, GRPO, root cause analysis, DPO, Stage-1 SFT, small-scale GRPO

8.7 Decision framework: Which technique?

The section gives an SFT-data-quality-first framework for choosing among preference optimization techniques based on the task, quality target, and cost.

Key terms: SFT-data-quality gate, DPO, quality target, preference pairs, GRPO, RFT, verifiable rewards, RLHF, binary pairs, PPO complexity

8.8 Honest assessment: Failure modes and practical limits

8.8.1 The 95/4/1 breakdown

This section uses an illustrative 95/4/1 split to help enterprises choose between improving SFT data, using DPO for targeted gains, and pursuing more demanding reward-based methods.

Key terms: 95/4/1 split, SFT data, target distribution, DPO, preference pairs, conciseness, SFT model, GRPO, RFT, verifiable domains

8.8.2 Known DPO failure modes

This section explains two DPO failure modes that can undermine enterprise deployments and how to detect or mitigate them.

Key terms: DPO, length and verboseness bias, output length distributions, length-controlled evaluation, SimPO, length-desensitization techniques, likelihood displacement, safety-related preference pairs, chosen response, refusal rates

8.8.3 Common mistakes

This section identifies common enterprise DPO mistakes involving the SFT foundation, preference-pair quality, the reference model, and factual accuracy.

Key terms: SFT, reference model, frozen pre-DPO checkpoint, preference pairs, pair quality, factual accuracy, hallucinations, retrieval augmentation, factuality-focused preference data

8.9 Preference optimization and safety

This section explains how DPO can affect safety, how to distinguish a new safety regression from an existing SFT weakness, and how to respond before release.

Key terms: DPO, helpfulness, refusal, likelihood displacement, safety regression testing, safety regression, SFT model, DPO model, release gate, refusal rate

8.9.1 Dual-objective optimization

This section explains how to keep preference optimization from weakening safety and how to test for safety regressions after model adaptation.

Key terms: dual-objective optimization, task quality, safety, dual-objective DPO, safety test suite, harmful request refusal, uncertainty acknowledgment, bias neutrality, general knowledge retention, safety regression

8.10 Summary

8.11 References

This section lists sources cited in the chapter without explaining additional concepts or procedures.

Where this is used later

Concepts explained in this chapter

← Chapter 7: Knowledge distillation: Capturing frontier model intelligence Chapter 9: Managing model evolution, drift, and versioning →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.