Preference
A preference is a comparative judgment or inclination that favors one alternative, outcome, or characteristic over others.
Reinforcement Learning from Human Feedback
A human preference is a person's comparative judgment about the desirability or quality of model behavior or outputs.
- How it is used
- Uses preferences to supply feedback for optimizing model behavior when rewards are hard to specify directly.
- How deep it goes
- Explains preference modeling and its theoretical limitations, while deferring the mechanics of collecting and structuring preference data.
- Examples
- Uses goldfish poems to contrast preference with correctness and cites summarization, instruction following, and dialogue as applications.
- What you can do after
- Evaluate/understand how preference judgments become proxy rewards and why optimizing those rewards may not capture all human considerations.
Read how Reinforcement Learning from Human Feedback explains preference →
Bayesian Optimization in Action
A user preference is a person's relative desirability judgment over products, experiences, or other alternatives.
- How it is used
- Frames product recommendation and black-box optimization as learning a user's objective from pairwise choices.
- How deep it goes
- Defines an inaccessible preference objective and develops one-dimensional setup, comparison encoding, GP modeling, and BayesOpt querying.
- Examples
- Uses product ratings, production recommendation, and a Hawaiian-shirt search over the number of flowers.
- What you can do after
- Apply preference optimization by collecting comparisons, encoding outcome indices, and selecting new queries with BayesOpt.
Read how Bayesian Optimization in Action explains preference →
LLM Evaluation and Alignment, The Foundational Ideas
A human preference is a person's comparative judgment that one output or behavior is more desirable than another.
- How it is used
- Uses comparative judgments to supply a training signal when demonstrations or hand-crafted rewards are difficult.
- How deep it goes
- Explains the concurrent rollout, labeling, and reward-model loops, including Bradley–Terry scoring and cross-entropy training.
- Examples
- Illustrates preference learning with simulated robot backflips and Pong, and a harmful trade-off with evasive refusals.
- What you can do after
- Evaluate preference signals by inspecting labeler agreement, rubric clarity, and unwanted trade-offs.
Read how LLM Evaluation and Alignment, The Foundational Ideas explains preference →
Hidden Influences
A user's inclination toward particular content, items, topics, or qualities of recommendations.
- How it is used
- Uses preferences to assess personalized recommendations and their role in distributing attention.
- How deep it goes
- Explains interaction-based learning and discovery trade-offs, then examines why recommendation value is difficult to measure.
- Examples
- Uses unequal engagement with two accounts, interest in dogs, and an undiscovered interest in books to illustrate preference learning.
- What you can do after
- Evaluate how interaction-based recommendations can miss interests and why their value is difficult to measure.
Algorithms Every Programmer Should Know
A preference indicates which potential match a participant favors over another.
- How it is used
- Models how participants rank potential matches so matching procedures can make and assess assignments.
- How deep it goes
- Works through ranked lists in a matching example and notes explicit and inferred preferences in applications.
- Examples
- Uses student–university rankings and dating criteria to illustrate preferences.
- What you can do after
- Understand how ranked or inferred preferences shape proposals, acceptances, and matching choices.
Read how Algorithms Every Programmer Should Know explains preference →
Deep Learning with Python, Third Edition
The relative ranking of candidate responses by people according to what is more helpful or desirable.
- How it is used
- Uses human rankings to turn judgments about helpfulness into a training signal for language-model improvement.
- How deep it goes
- Defines ranking-based data collection and reward modeling, then sketches score-based filtering without deriving the training loss.
- Examples
- Uses ranked prompt responses and a generate-score-filter-fit loop as concrete illustrations.
- What you can do after
- Evaluate how human rankings can supervise a reward model and guide response selection.
Read how Deep Learning with Python, Third Edition explains preference →
Also covered in
- Collaborative Software Design: How to facilitate domain modeling decisions
- Effective Conversational AI
- Introduction to Generative AI, Second Edition
- AI Recommender SystemsIn early access (MEAP)
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
- AI-Powered Search
- Grokking AI Algorithms, Second Edition
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- promptA prompt is text or other input supplied to an AI model to provide instructions, context, or other information that guides its response or generated output.
- fine-tuningFine-tuning is the continued training of a pretrained model on targeted data or objectives to adapt its parameters or behavior to a task, domain, or use.
- actionAn action is a choice or operation selected or performed by an actor that can affect a system, environment, or process.
Related concepts
- RLHFused by · Reinforcement learning from human feedback is a model-training approach that uses human evaluations or preferences as feedback to optimize model behavior.
- reward modelused by · A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
- experimentationmeasured by · Experimentation is the planned investigation of a question or hypothesis by conducting trials or interventions and observing their outcomes.
- Chatbot Arenameasured by · Chatbot Arena is an online evaluation platform and leaderboard that compares conversational language models through users’ pairwise preference judgments of their responses.
- inner productmeasured by · An inner product is a scalar-valued operation on two vectors that generalizes the dot product, which for real coordinate vectors is the sum of products of corresponding components.
- pairwise comparisonused by · A pairwise comparison is an evaluation of two alternatives against one another to determine their relative preference, value, or rank.
Questions about preference
What is preference?
A preference is a comparative judgment or inclination that favors one alternative, outcome, or characteristic over others.
Which Manning books explain preference?
Reinforcement Learning from Human Feedback, Bayesian Optimization in Action, LLM Evaluation and Alignment, The Foundational Ideas, Hidden Influences, Algorithms Every Programmer Should Know and Deep Learning with Python, Third Edition.
What should I know before learning preference?
Manning books that teach preference usually introduce language model, prompt, fine-tuning and action first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





