Direct Preference Optimization

As explained in Transformers in Action

Direct Preference Optimization is a preference-alignment technique that directly optimizes a language model from chosen and rejected responses without separately training an explicit reward model.

Direct Preference Optimization (DPO) aligns a language model with human preferences by training it to favor a chosen response over a rejected one for the same prompt. Unlike reinforcement learning from human feedback (RLHF), it does not first train an explicit reward model and then use reinforcement learning to optimize against that model. section 5.2

Figure 5.1 A simplified comparison of RLHF and DPO and how DPO directly optimizes for the policy without an explicit reward function

The comparison shows the central workflow difference: DPO optimizes the policy directly rather than introducing a separate reward-model stage. section 5.2

Training uses preference-labeled prompt–response pairs after supervised fine-tuning. The DPO step takes the prompt into account, promotes the chosen response, and deprioritizes the rejected response. Kullback–Leibler (KL) divergence is part of the constraint-based optimization; the preference decision can be understood as a classification-like task. section 5.2

A practical training sequence is:

• Load a previously trained model and add LoRA adapters. Select preference samples; the worked setup samples 1% of UltraFeedback Binarized’s train_prefs and test_prefs splits. section 5.2.2
• Transform chat messages into the prompt, chosen, and rejected fields expected by DPOTrainer. Keep those three fields when mapping the dataset. section 5.2.2
• Configure training arguments, pass the transformed training and evaluation splits to TRL’s DPOTrainer, call train(), and save the model locally or to Hugging Face Hub. section 5.2.2

Listing 5.16 Preparing training data

transformed_datasets = raw_datasets.map(
    lambda example: apply_chat_template(example, tokenizer),
    remove_columns=[col for col in raw_datasets["train_prefs"].column_names 
    if col not in ['chosen', 'rejected', 'prompt']],
    desc="Formatting prompt template",
)

The dataset mapping shows where chat formatting happens and which columns survive for DPO training. section 5.2.2

Listing 5.18 Initializing training arguments for DPO training

dpo_trainer = DPOTrainer(
    model=model,
    ref_model=None,  
    args=training_args,
    beta=0.1,
    train_dataset=transformed_datasets["train_prefs"],
    eval_dataset=transformed_datasets["test_prefs"],
    tokenizer=tokenizer,
    max_length=1024,
    max_prompt_length=512,
)

# Continue with your training process
longcomment/1/Train the model with DPO/
dpo_trainer.train()

The trainer setup makes the applied choices concrete: beta=0.1, separate training and evaluation splits, a tokenizer, and prompt and sequence length limits. section 5.2.2

DPO’s simpler workflow does not remove every training risk. Its KL-based loss can overfit with deterministic preferences or limited datasets, potentially producing an overly deterministic policy. The trainer offers loss_type="ipo" as a regularization-oriented alternative and loss_type="kto_pair" for Kahneman–Tversky optimization. Model, dataset, and hyperparameter choices remain useful experiments rather than fixed defaults. section 5.2.4

How Transformers in Action covers it

How deep it goes

Explains preference pairs and KL divergence, then gives dataset preparation and trainer steps; stops short of deriving the loss.

Examples

Uses UltraFeedback Binarized preference splits, chat-template transformation, DPOTrainer configuration, and local or Hub model saving.

What it assumes

Assumes a previously trained model and tokenizer; introduces preference labeling and the DPO training sequence.

What you can do after

Apply it via TRL’s DPOTrainer to fine-tune a previously trained model on formatted preference data.

How it connects

Connects DPO to supervised fine-tuning, RLHF, KL divergence, LoRA, and alternative preference losses.

liveBookmanning.com

Read Transformers in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackDirect preference optimization (DPO) directly trains a policy from pairwise preferred and dispreferred outputs relative to a reference policy.

Compare how 2 books explain Direct Preference Optimization →

Related concepts

• reinforcement learning with human feedbackcontrasts with · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.
• preference datauses · Preference data is a collection of judgments or comparisons indicating the relative preference or quality of alternatives, often model outputs.
• Group Relative Policy Optimizationequivalent to · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled outputs or actions.
• preference tuningused by · Preference tuning is a model-training technique that adjusts a model’s behavior toward outputs favored by preference signals, judgments, objectives, or rewards.

This explainer was generated by Manning's knowledge graph from the text of Transformers in Action. The section links point to the parts of the book it draws on.