A chosen response is the preferred answer in a preference pair: the model is trained to favor it over a rejected response to the same prompt. Preferences can come from human judgments or automated systems. section 8.2 section 8.2.5
A preference record contains:
- A prompt, represented as the conversation leading up to the user’s question. section 8.2.1
- A chosen assistant response and a rejected assistant response to that prompt. section 8.2.1
Enterprise support preference
For an IT installation question, prefer the answer that follows the organization’s format and directs users to its approved Software Center. A fluent answer directing users to a public download can be rejected, even if technically correct. section 8.2.1
Figure 8.3 The DPO training mechanism: raise the chosen response, lower the rejected one, stay near the frozen reference.

The diagram shows how DPO raises the chosen response’s probability while lowering the rejected response’s and constraining drift from a frozen reference. section 8.2.5
