liveBook · Manning

DPOTrainer

As explained in LLM Customization and Fine-Tuning

A TRL trainer for optimizing transformer language models from preference pairs using Direct Preference Optimization.

DPOTrainer, from TRL, trains a language model on preference pairs rather than individual examples. It refines an SFT-capable model’s preferences while keeping it close to a frozen reference model. section 8.2.3

A practical starting configuration balances preference learning against drift, overfitting, and memory:

  • Use a learning rate of 5e-6, beta 0.1, one epoch, and gradient checkpointing. Higher learning rates risk capability loss; too little or too much beta risks mode collapse or minimal change. section 8.2.3
  • Plan for more memory than SFT: full-parameter DPO of a 4B model peaked near 54 GB across three cards and did not fit on one 24 GB card at the stated sequence length. LoRA is offered as a single-GPU alternative. section 8.2.3

A small preference run

A run starts from an SFT checkpoint and uses 270 training pairs and 30 validation pairs. With beta 0.1, learning rate 5e-6, and one epoch, reported held-out reward accuracy is 0.600. section 8.2.3

Listing 8.2 train_dpo.py (key sections)
# (key sections -- full source in code/chapter08/train_dpo.py)
 
from trl import DPOConfig, DPOTrainer
 
def main() -> None:
    # ...
 
    tokenizer = AutoTokenizer.from_pretrained(args.model_dir,  #A
                                      trust_remote_code=True)  #A
    if tokenizer.pad_token_id is None:
        tokenizer.pad_token = tokenizer.eos_token
    tokenizer.padding_side = "right"
 
    model = AutoModelForCausalLM.from_pretrained(
        args.model_dir,
        torch_dtype=torch.bfloat16,
        device_map="auto",
        trust_remote_code=True,
    )
    
    train_ds = load_preference_dataset(args.train)     #B
    valid_ds = load_preference_dataset(args.valid)     #B
 
    dpo_config = DPOConfig(
        output_dir=str(out_dir),
        num_train_epochs=args.epochs,                   #C
        per_device_train_batch_size=args.batch_size,    #C
        gradient_accumulation_steps=args.grad_accum,    #C
        learning_rate=args.lr,                          #C
        beta=args.beta,                                 #C
        max_length=args.max_length,                     #C
        max_prompt_length=args.max_prompt_length,       #C
        warmup_ratio=0.1,
        eval_strategy="epoch",
        save_strategy="epoch",
        save_total_limit=1,
        load_best_model_at_end=True,
        metric_for_best_model="eval_loss",
        bf16=use_bf16,
        gradient_checkpointing=True,
    )
 
    # DPO trainer
    trainer = DPOTrainer(
        model=model,
        args=dpo_config,
        train_dataset=train_ds,
        eval_dataset=valid_ds,
        processing_class=tokenizer,
    )
 
    trainer.train()
    trainer.save_model(str(out_dir))
    tokenizer.save_pretrained(str(out_dir))

Inspect the Python listing for the TRL imports, model and tokenizer setup, preference-dataset loading, and DPOConfig scaffold. section 8.2.3

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains key hyperparameter tradeoffs and memory demands, but does not detail trainer internals.
Examples
Uses a 270-pair training set, 30-pair validation set, and a 4B-model hardware example.
What it assumes
Assumes an SFT checkpoint and preference pairs; does not explain how to create either.
What you can do after
Apply TRL’s DPOTrainer to refine an SFT checkpoint using preference pairs and practical training settings.
How it connects
Connects Direct Preference Optimization with TRL, SFTTrainer, supervised fine-tuning, KL penalties, and LoRA.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.