A human preference is a comparative judgment that one model behavior or output is more desirable or higher quality than another. It supplies a training signal when the intended behavior cannot be specified cleanly as a programmatic reward, especially in human-facing content and interaction. section 1.0 section 10.0
In RLHF, preferences turn judgments over outputs into an optimizable proxy. A reward model compresses preference relations between text completions into one scalar score; reinforcement learning then adjusts a model to produce completions that receive higher scores. Instruction fine-tuning first establishes a question-answering format, while the later reward-model and RL stages target answer quality. section 1.3 section 5.9
The operational pipeline is:
- Collect prompts, multiple model completions, and judgments identifying which completion is better; diversity of prompts and labelers is used to seek a more robust signal. section 1.3
- Train a reward model, commonly a pretrained language model with a linear scalar head, to predict which completion is preferred. Standard reward models use a Bradley–Terry formulation and contrastive loss. section 5.9
- Generate candidate completions, score or rank them with the reward model, and use RL to shift model parameters toward higher-scoring token sequences. section 1.3
- Constrain optimization so the model does not change too drastically; early language-model RLHF established KL divergence as this regularizer. section 2.2
Preference versus correctness
Two poems about a goldfish can be compared by asking which is better and which model produced it, without an objectively correct answer. This differs from asking who is president of the United States, where correctness provides a clearer target. Preference labels provide an indirect metric for the former kind of task. section 10.1
A scalar preference score is useful but not a complete account of what people value. Human judgments can be noisy, context-dependent, temporally shifting, and non-transitive; sequential labeling can itself shift preferences. Reward modeling therefore compresses changing, multidimensional considerations into a function approximation rather than recovering one true reward. section 10.5 section 10.6
This mismatch creates practical failure modes. RL was designed around stable, determinate rewards, whereas preferences may vary across people and situations. Optimizing a reward model too aggressively can overfit to its learned proxy, producing effects such as excessive length or chattiness and harming performance elsewhere. Practical RLHF consequently focuses on tractable style and performance targets rather than resolving the full complexity of human values. section 1.7 section 10.6
Preference-based training has been used for language-model summarization, instruction following, browser-assisted question answering, citation-supported answers, and dialogue. Related reward-model variants target reasoning differently: outcome reward models predict correctness per token, while process reward models score individual reasoning steps. section 2.2 section 5.9
