Preference tuning

As explained in Reinforcement Learning from Human Feedback

Preference optimization is model training that adjusts behavior toward preferred outputs using preference-based objectives or rewards.

Preference tuning, also called preference fine-tuning (PreFT), is the post-training stage that adjusts a language model toward outputs people prefer. It follows instruction fine-tuning, which establishes instruction-following format and basic behavior, and targets subtler qualities of language that are difficult to specify as an ordinary loss function. RLHF is the established core of this area, while newer direct and reward-based approaches broaden it beyond a single recipe. section 1.1 section 2.3

The central difficulty is that desired human behavior cannot be precisely written as a reward or loss function. Preference data therefore acts as a proxy signal: labeled comparisons help a model match desired behaviors and avoid unwanted ones. This makes data collection a technical and operational part of optimization, not merely an annotation step; collection is costly, complex, and often needs generations from the model currently being trained. section 11.0 section 11.2.5

Common formulations and placement in a training pipeline include:

• RLHF: train a supervised model, build a reward model from preference data, then optimize the model with reinforcement learning. section 1.1
• Direct preference optimization (DPO): take gradient steps directly on pairwise preference data, pursuing the RLHF optimization problem with fewer moving parts. section 1.5
• Process rewards and execution feedback: extend preference-oriented training to intermediate reasoning steps and feedback from code, math, or other online reasoning settings. section 2.3
• Tool-use preference optimization: compare trajectories to improve the decision of whether to call a tool or answer directly; use environment-feedback RL for multistep task success. section 13.5 section 13.6

Choosing whether to call a tool

For a tool-use prompt, supervised fine-tuning can teach the syntax and selection of a function call. Preference optimization over complete trajectories can then favor an appropriate tool call over answering directly. When several tool actions are required, task success and constraint satisfaction provide environment feedback for reinforcement learning. section 13.5 section 13.6

Figure 1.1 A rendition of the early three-stage RLHF process: training via supervised fine-tuning (SFT; chapter 4), building a reward model (RM, chapter 5), and then optimizing with RL (chapter 6).

The three-stage RLHF diagram situates preference tuning between instruction-oriented supervised fine-tuning and reinforcement-learning optimization through a learned reward model. section 1.1

In open post-training, DPO became influential because it simplified the optimization path, but strong results still depended on practical choices such as a lower learning rate and suitable data. Preference tuning is not a complete post-training recipe: instruction fine-tuning, RLHF or direct methods, prompt design, and newer verifiable-reward training can be combined in different orders for different capabilities. section 1.5 section 1.1

How Reinforcement Learning from Human Feedback covers it

How deep it goes

Explains RLHF stages, direct pairwise optimization, data constraints, and tool-use applications without prescribing a universal recipe.

Examples

Uses an RLHF three-stage pipeline and tool trajectories that choose tool calls versus direct answers.

What it assumes

Assumes familiarity with post-training stages while motivating preference data as a substitute for explicit human reward functions.

What you can do after

Apply it via RLHF or DPO-style objectives when preference data can proxy desired model behavior.

How it connects

Connects RLHF, instruction fine-tuning, DPO, preference data, reward models, RLVR, and tool use.

liveBookmanning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Reasoning Model (From Scratch)A post-training method that adjusts a model to produce responses aligned with preferred outputs or styles.

Compare how 2 books explain preference tuning →

Related concepts

• Direct Preference Optimizationuses · Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.
• reinforcement learning with human feedbackuses · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.
• Group Relative Policy Optimizationuses · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled outputs or actions.
• preference dataused by · Preference data is a collection of judgments or comparisons indicating the relative preference or quality of alternatives, often model outputs.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.