liveBook · Manning

Preference tuning

As explained in LLM Customization and Fine-Tuning

Training a model from preference signals so that it favors desired responses over less desired ones.

A model can give a factually correct answer and still be too verbose, poorly structured, or inconsistent in tone. Preference tuning changes the model’s behavior using signals about which responses are better, rather than relying only on examples of acceptable output. section 8.0 section 8.1

Supervised fine-tuning (SFT) establishes task competence, format, and domain vocabulary. Preference tuning then refines choices among plausible responses—for example, favoring concise over verbose answers or balancing safety with usefulness. The preference signal works best when the starting model can already produce responses in the right format. section 6.10 section 6.10.1

Direct Preference Optimization (DPO) trains on preferred and dispreferred responses to the same prompt. It shifts the model’s own generations toward the preferred behavior; it does not turn the model into a separate response-scoring judge. Preference pairs can be human-graded or generated by contrasting stronger and weaker models. section 8.1 section 8.10

Choose the signal and technique for the problem:

  • Fix noisy, inconsistent, or unrepresentative SFT data first. If the model already meets the quality bar, preference tuning may not be needed. section 8.8.1 section 6.10
  • Use DPO for targeted preferences such as tone or conciseness. It avoids the separate reward model and infrastructure complexity of full reinforcement learning from human feedback (RLHF). section 1.6.2 section 8.10
  • Consider Group Relative Policy Optimization (GRPO) or reinforcement fine-tuning when outcomes can be checked programmatically, as in code, math, or structured extraction. Reserve more expensive approaches for requirements that justify them. section 8.8.1 section 8.10

Conciseness in an IT help desk

A DPO run used 130 preference pairs on an SFT model to target conciseness. Answers were 21% shorter, and a blinded judge preferred them 61% of the time; Token-F1 stayed approximately flat, moving from 0.168 to 0.166. The result illustrates a targeted style gain, not a general improvement in objective correctness. section 8.10 section 8.8.1

Evaluate base, SFT, and preference-tuned models on a held-out set representative of the production workload. Test safety after every preference optimization step: improving helpfulness can weaken refusal boundaries. Monitor for length or verboseness bias and likelihood displacement after deployment. section 1.6.2 section 8.0 section 8.10

How LLM Customization and Fine-Tuning covers it

How deep it goes
Walks through preference-pair construction, DPO training, evaluation, safety testing, and alternative technique tradeoffs.
Examples
Uses an IT help-desk conciseness run and the DeepSeek-R1 cold-start pattern to illustrate targeted preference gains and SFT followed by GRPO.
What it assumes
Assumes a model that already produces responses in the correct format and uses an SFT checkpoint as the preference-training starting point.
What you can do after
Apply DPO via TRL’s DPOTrainer and evaluate preference-tuned models, including safety regression.
How it connects
Connects preference tuning to supervised fine-tuning, direct preference optimization, reinforcement learning from human feedback, and group relative policy optimization.

Also explained in

Compare how 3 books explain preference tuning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.