DPOTrainer, from TRL, trains a language model on preference pairs rather than individual examples. It refines an SFT-capable model’s preferences while keeping it close to a frozen reference model. section 8.2.3
A practical starting configuration balances preference learning against drift, overfitting, and memory:
- Use a learning rate of 5e-6, beta 0.1, one epoch, and gradient checkpointing. Higher learning rates risk capability loss; too little or too much beta risks mode collapse or minimal change. section 8.2.3
- Plan for more memory than SFT: full-parameter DPO of a 4B model peaked near 54 GB across three cards and did not fit on one 24 GB card at the stated sequence length. LoRA is offered as a single-GPU alternative. section 8.2.3
A small preference run
A run starts from an SFT checkpoint and uses 270 training pairs and 30 validation pairs. With beta 0.1, learning rate 5e-6, and one epoch, reported held-out reward accuracy is 0.600. section 8.2.3
Listing 8.2 train_dpo.py (key sections)
# (key sections -- full source in code/chapter08/train_dpo.py)
from trl import DPOConfig, DPOTrainer
def main() -> None:
# ...
tokenizer = AutoTokenizer.from_pretrained(args.model_dir, #A
trust_remote_code=True) #A
if tokenizer.pad_token_id is None:
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
model = AutoModelForCausalLM.from_pretrained(
args.model_dir,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
)
train_ds = load_preference_dataset(args.train) #B
valid_ds = load_preference_dataset(args.valid) #B
dpo_config = DPOConfig(
output_dir=str(out_dir),
num_train_epochs=args.epochs, #C
per_device_train_batch_size=args.batch_size, #C
gradient_accumulation_steps=args.grad_accum, #C
learning_rate=args.lr, #C
beta=args.beta, #C
max_length=args.max_length, #C
max_prompt_length=args.max_prompt_length, #C
warmup_ratio=0.1,
eval_strategy="epoch",
save_strategy="epoch",
save_total_limit=1,
load_best_model_at_end=True,
metric_for_best_model="eval_loss",
bf16=use_bf16,
gradient_checkpointing=True,
)
# DPO trainer
trainer = DPOTrainer(
model=model,
args=dpo_config,
train_dataset=train_ds,
eval_dataset=valid_ds,
processing_class=tokenizer,
)
trainer.train()
trainer.save_model(str(out_dir))
tokenizer.save_pretrained(str(out_dir))Inspect the Python listing for the TRL imports, model and tokenizer setup, preference-dataset loading, and DPOConfig scaffold. section 8.2.3
