Preference tuning
Preference tuning is a model-training technique that adjusts a model’s behavior toward outputs favored by preference signals, judgments, objectives, or rewards.
LLM Customization and Fine-Tuning
Training a model from preference signals so that it favors desired responses over less desired ones.
- How it is used
- Refines response quality and alignment after a model has learned the task, especially when acceptable answers differ in style, helpfulness, or safety.
- How deep it goes
- Walks through preference-pair construction, DPO training, evaluation, safety testing, and alternative technique tradeoffs.
- Examples
- Uses an IT help-desk conciseness run and the DeepSeek-R1 cold-start pattern to illustrate targeted preference gains and SFT followed by GRPO.
- What you can do after
- Apply DPO via TRL’s DPOTrainer and evaluate preference-tuned models, including safety regression.
Read how LLM Customization and Fine-Tuning explains preference tuning →
Reinforcement Learning from Human Feedback
Preference optimization is model training that adjusts behavior toward preferred outputs using preference-based objectives or rewards.
- How it is used
- Uses preference tuning to align language-model behavior with hard-to-quantify human preferences during post-training.
- How deep it goes
- Compares reward-model-based RLHF with direct optimization on preference pairs, without deriving their objectives.
- Examples
- Illustrates preference optimization through tool trajectories that distinguish calling a tool from answering directly.
- What you can do after
- Evaluate preference-tuning approaches and data constraints when planning language-model post-training.
Read how Reinforcement Learning from Human Feedback explains preference tuning →
Build a Reasoning Model (From Scratch)
A post-training method that adjusts a model to produce responses aligned with preferred outputs or styles.
- How it is used
- Positions preference tuning after instruction tuning to refine response style and align outputs with user preferences.
- How deep it goes
- Explains RLHF through rankings, reward-model training, and RL updates without detailing the complete optimization algorithm.
- Examples
- Uses ranked laptop recommendations and staged sleep-health summaries to illustrate specificity, tone, and user-centered responses.
- What you can do after
- Evaluate/understand how human rankings become reward signals that steer a language model toward preferred whole responses.
Read how Build a Reasoning Model (From Scratch) explains preference tuning →
What to know first
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- reward signalA reward signal is a value or feedback indicating the desirability, benefit, or cost associated with an action, outcome, or behavior.
- foundation modelA foundation model is a broadly trained machine-learning model whose learned capabilities can be reused or adapted for a range of downstream tasks.
Related concepts
- Direct Preference Optimizationuses · Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.
- Group Relative Policy Optimizationuses · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled actions or outputs.
- RLHFuses · Reinforcement learning from human feedback is a model-training approach that uses human evaluations or preferences as feedback to optimize model behavior.
- supervised fine-tuningcontrasts with · Supervised fine-tuning is further training a pretrained model on labeled input-output examples to adapt its behavior to desired outputs.
- checkpointingused by · Checkpointing is the recording of an intermediate state or progress point in a process to support recovery, resumption, inspection, or control.
- preference dataused by · Preference data is a collection of judgments about the relative preference or quality of alternatives, often candidate outputs for the same input.
Questions about preference tuning
What is preference tuning?
Preference tuning is a model-training technique that adjusts a model’s behavior toward outputs favored by preference signals, judgments, objectives, or rewards.
Which Manning books explain preference tuning?
LLM Customization and Fine-Tuning, Reinforcement Learning from Human Feedback and Build a Reasoning Model (From Scratch).
What should I know before learning preference tuning?
Manning books that teach preference tuning usually introduce fine-tuning, language model, reward signal and foundation model first.
What is the difference between preference tuning and supervised fine-tuning?
Preference tuning is a model-training technique that adjusts a model’s behavior toward outputs favored by preference signals, judgments, objectives, or rewards. Supervised fine-tuning is further training a pretrained model on labeled input-output examples to adapt its behavior to desired outputs.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


