Preference
A preference is a comparative judgment or inclination that favors one alternative, outcome, or characteristic over others.
Reinforcement Learning from Human Feedback
A human preference is a person's comparative judgment about the desirability or quality of model behavior or outputs.
- How it is used
- Supplies comparative feedback for optimizing model behavior when target quality is hard to specify programmatically.
- How deep it goes
- Explains collection, scalar reward modeling, RL optimization, and limits from unstable, context-dependent judgments.
- Examples
- Contrasts goldfish-poem comparisons with factual questions and traces applications in summarization, instruction following, question answering, and dialogue.
- What you can do after
- Evaluate/understand why preference signals enable RLHF yet can misrepresent changing human judgments and reward-model proxies.
Read how Reinforcement Learning from Human Feedback explains preference →
Bayesian Optimization in Action
A user preference is a person's relative desirability judgment over products, experiences, or other alternatives.
- How it is used
- Frames product recommendation and black-box optimization as learning a user's objective from pairwise choices.
- How deep it goes
- Defines an inaccessible preference objective and develops one-dimensional setup, comparison encoding, GP modeling, and BayesOpt querying.
- Examples
- Uses product ratings, production recommendation, and a Hawaiian-shirt search over the number of flowers.
- What you can do after
- Apply preference optimization by collecting comparisons, encoding outcome indices, and selecting new queries with BayesOpt.
Read how Bayesian Optimization in Action explains preference →
Deep Learning with Python, Third Edition
The relative ranking of candidate responses by people according to what is more helpful or desirable.
- How it is used
- Uses human rankings to turn judgments about helpfulness into a training signal for language-model improvement.
- How deep it goes
- Defines ranking-based data collection and reward modeling, then sketches score-based filtering without deriving the training loss.
- Examples
- Uses ranked prompt responses and a generate-score-filter-fit loop as concrete illustrations.
- What you can do after
- Evaluate how human rankings can supervise a reward model and guide response selection.
Read how Deep Learning with Python, Third Edition explains preference →
Related concepts
- reinforcement learning with human feedbackused by · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.
- reward modelused by · A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
- recommender systemused by · A recommender system is software that predicts, selects, or ranks items, content, or actions for users based on available information about users, items, or context.
- experimentationmeasured by · Experimentation is the deliberate testing of an idea, intervention, or alternative by observing outcomes to gather evidence or compare effects.
- user profilepart of · A user profile is a collection of information associated with a user or account, such as identifying attributes, preferences, settings, or activity.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.