liveBook · Manning

Rejected response

As explained in LLM Customization and Fine-Tuning

The candidate response in a preference pair that is less preferred than its paired alternative.

A rejected response is the less-preferred answer in a preference pair. In DPO, training lowers its probability relative to the chosen answer while a frozen reference model constrains how far the policy changes. section 8.2 section 8.2.5

Enterprise support preference

For a development-tool installation question, a response directing the user to an unapproved public download can be rejected in favor of one that follows the organization’s steps through its Software Center—even when both answers are technically correct. section 8.2.1

Figure 8.3 The DPO training mechanism: raise the chosen response, lower the rejected one, stay near the frozen reference.

The diagram shows the rejected response’s role in DPO alongside the chosen response and frozen reference. section 8.2.5

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines the response by relative preference and explains how DPO lowers its probability while anchoring updates to a frozen reference.
Examples
Contrasts approved Software Center installation guidance with a technically correct answer pointing to a public download.
What you can do after
Understand how to pair a less-preferred answer with a chosen response for DPO training.
How it connects
Connects rejected responses to DPO, preference pairs, chosen responses, and reference models.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.