liveBook · Manning

Preference tuning

Preference tuning is a model-training technique that adjusts a model’s behavior toward outputs favored by preference signals, judgments, objectives, or rewards.

How 3 Manning books explain it

LLM Customization and Fine-Tuning

Training a model from preference signals so that it favors desired responses over less desired ones.

How it is used
Refines response quality and alignment after a model has learned the task, especially when acceptable answers differ in style, helpfulness, or safety.
How deep it goes
Walks through preference-pair construction, DPO training, evaluation, safety testing, and alternative technique tradeoffs.
Examples
Uses an IT help-desk conciseness run and the DeepSeek-R1 cold-start pattern to illustrate targeted preference gains and SFT followed by GRPO.
What you can do after
Apply DPO via TRL’s DPOTrainer and evaluate preference-tuned models, including safety regression.

Read how LLM Customization and Fine-Tuning explains preference tuning →

Reinforcement Learning from Human Feedback

Preference optimization is model training that adjusts behavior toward preferred outputs using preference-based objectives or rewards.

How it is used
Uses preference tuning to align language-model behavior with hard-to-quantify human preferences during post-training.
How deep it goes
Compares reward-model-based RLHF with direct optimization on preference pairs, without deriving their objectives.
Examples
Illustrates preference optimization through tool trajectories that distinguish calling a tool from answering directly.
What you can do after
Evaluate preference-tuning approaches and data constraints when planning language-model post-training.

Read how Reinforcement Learning from Human Feedback explains preference tuning →

Build a Reasoning Model (From Scratch)

A post-training method that adjusts a model to produce responses aligned with preferred outputs or styles.

How it is used
Positions preference tuning after instruction tuning to refine response style and align outputs with user preferences.
How deep it goes
Explains RLHF through rankings, reward-model training, and RL updates without detailing the complete optimization algorithm.
Examples
Uses ranked laptop recommendations and staged sleep-health summaries to illustrate specificity, tone, and user-centered responses.
What you can do after
Evaluate/understand how human rankings become reward signals that steer a language model toward preferred whole responses.

Read how Build a Reasoning Model (From Scratch) explains preference tuning →

What to know first

Related concepts

Questions about preference tuning

What is preference tuning?

Preference tuning is a model-training technique that adjusts a model’s behavior toward outputs favored by preference signals, judgments, objectives, or rewards.

Which Manning books explain preference tuning?

LLM Customization and Fine-Tuning, Reinforcement Learning from Human Feedback and Build a Reasoning Model (From Scratch).

What should I know before learning preference tuning?

Manning books that teach preference tuning usually introduce fine-tuning, language model, reward signal and foundation model first.

What is the difference between preference tuning and supervised fine-tuning?

Preference tuning is a model-training technique that adjusts a model’s behavior toward outputs favored by preference signals, judgments, objectives, or rewards. Supervised fine-tuning is further training a pretrained model on labeled input-output examples to adapt its behavior to desired outputs.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.