liveBook · Manning

Contrastive loss function

As explained in Reinforcement Learning from Human Feedback

A contrastive loss is an optimization objective that compares examples to encourage similar or preferred examples to be distinguished from dissimilar or non-preferred ones.

A contrastive loss function trains from comparisons rather than scoring each example independently. In preference-based language-model training, it distinguishes a better completion from a worse one for the same prompt, supplying both positive and negative feedback. This differs from instruction fine-tuning, which updates next-token predictions toward a target response. section 1.2

The practical objective is to make preferred outputs score or rank above rejected outputs. A standard RLHF reward model uses a Bradley–Terry preference formulation and contrastive training to predict the probability that one completion is preferred over another. The reward model compresses human judgments into a scalar signal that later optimization can use. section 5.9

Contrastive training changes the data and optimization pipeline in several ways:

  • Each training item needs a comparison, commonly a chosen completion and a rejected completion, rather than only a single desired answer. section 1.2 section 8.4
  • The signal is response-level: it judges which complete response is better, instead of directly prescribing every next token. section 1.2
  • For direct-alignment algorithms, the loss encourages separation between chosen and rejected completions; the difference between them may matter more than the exact models that generated them. section 8.4
  • Synthetic pairwise data can provide a simple, high-performing starting point, but judge-model biases can affect which responses enter the chosen side of a pair. section 8.4

Preference pair for a prompt

Given one prompt, collect two completions and mark one as chosen and the other as rejected. A contrastive objective trains the system to assign the chosen completion a higher preference than the rejected one. In a reward-model pipeline, a linear head on a pretrained language model can emit one scalar score for each completion before the pair is compared. section 5.9 section 8.4

Contrastive objectives offer flexibility because they communicate what to prefer and avoid without requiring an exact target response. They also require careful control: reward signals are proxy objectives, can be over-optimized, and may inherit length or stylistic biases from preference data or automated judges. section 1.2 section 8.4

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Defines comparison-based training and links it to Bradley–Terry reward modeling, without specifying a full loss equation.
Examples
Uses chosen and rejected completions, including synthetic pairs generated by stronger and weaker models.
What it assumes
Assumes familiarity with language-model completions and introduces comparison-based preference training directly.
What you can do after
Apply it via reward models or direct-alignment algorithms to train from pairwise completion preferences.
How it connects
Connects RLHF, instruction fine-tuning, reward models, Bradley–Terry models, direct-alignment algorithms, and synthetic preference data.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.