Preference tuning
Preference tuning is a model-training technique that adjusts a model’s behavior toward outputs favored by preference signals, judgments, objectives, or rewards.
Reinforcement Learning from Human Feedback
Preference optimization is model training that adjusts behavior toward preferred outputs using preference-based objectives or rewards.
- How it is used
- Aligns post-trained language models to human preferences after instruction fine-tuning.
- How deep it goes
- Explains RLHF stages, direct pairwise optimization, data constraints, and tool-use applications without prescribing a universal recipe.
- Examples
- Uses an RLHF three-stage pipeline and tool trajectories that choose tool calls versus direct answers.
- What you can do after
- Apply it via RLHF or DPO-style objectives when preference data can proxy desired model behavior.
Read how Reinforcement Learning from Human Feedback explains preference tuning →
Build a Reasoning Model (From Scratch)
A post-training method that adjusts a model to produce responses aligned with preferred outputs or styles.
- How it is used
- Positions preference tuning after instruction tuning to refine response style and align outputs with user preferences.
- How deep it goes
- Explains RLHF through rankings, reward-model training, and RL updates without detailing the complete optimization algorithm.
- Examples
- Uses ranked laptop recommendations and staged sleep-health summaries to illustrate specificity, tone, and user-centered responses.
- What you can do after
- Evaluate/understand how human rankings become reward signals that steer a language model toward preferred whole responses.
Read how Build a Reasoning Model (From Scratch) explains preference tuning →
Related concepts
- Direct Preference Optimizationuses · Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.
- reinforcement learning with human feedbackuses · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.
- Group Relative Policy Optimizationuses · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled outputs or actions.
- preference dataused by · Preference data is a collection of judgments or comparisons indicating the relative preference or quality of alternatives, often model outputs.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.