A model can give a factually correct answer and still be too verbose, poorly structured, or inconsistent in tone. Preference tuning changes the model’s behavior using signals about which responses are better, rather than relying only on examples of acceptable output. section 8.0 section 8.1
Supervised fine-tuning (SFT) establishes task competence, format, and domain vocabulary. Preference tuning then refines choices among plausible responses—for example, favoring concise over verbose answers or balancing safety with usefulness. The preference signal works best when the starting model can already produce responses in the right format. section 6.10 section 6.10.1
Direct Preference Optimization (DPO) trains on preferred and dispreferred responses to the same prompt. It shifts the model’s own generations toward the preferred behavior; it does not turn the model into a separate response-scoring judge. Preference pairs can be human-graded or generated by contrasting stronger and weaker models. section 8.1 section 8.10
Choose the signal and technique for the problem:
- Fix noisy, inconsistent, or unrepresentative SFT data first. If the model already meets the quality bar, preference tuning may not be needed. section 8.8.1 section 6.10
- Use DPO for targeted preferences such as tone or conciseness. It avoids the separate reward model and infrastructure complexity of full reinforcement learning from human feedback (RLHF). section 1.6.2 section 8.10
- Consider Group Relative Policy Optimization (GRPO) or reinforcement fine-tuning when outcomes can be checked programmatically, as in code, math, or structured extraction. Reserve more expensive approaches for requirements that justify them. section 8.8.1 section 8.10
Conciseness in an IT help desk
A DPO run used 130 preference pairs on an SFT model to target conciseness. Answers were 21% shorter, and a blinded judge preferred them 61% of the time; Token-F1 stayed approximately flat, moving from 0.168 to 0.166. The result illustrates a targeted style gain, not a general improvement in objective correctness. section 8.10 section 8.8.1
Evaluate base, SFT, and preference-tuned models on a held-out set representative of the production workload. Test safety after every preference optimization step: improving helpfulness can weaken refusal boundaries. Monitor for length or verboseness bias and likelihood displacement after deployment. section 1.6.2 section 8.0 section 8.10


