liveBook · Manning

Preference

A preference is a comparative judgment or inclination that favors one alternative, outcome, or characteristic over others.

How 6 Manning books explain it

Reinforcement Learning from Human Feedback

A human preference is a person's comparative judgment about the desirability or quality of model behavior or outputs.

How it is used
Uses preferences to supply feedback for optimizing model behavior when rewards are hard to specify directly.
How deep it goes
Explains preference modeling and its theoretical limitations, while deferring the mechanics of collecting and structuring preference data.
Examples
Uses goldfish poems to contrast preference with correctness and cites summarization, instruction following, and dialogue as applications.
What you can do after
Evaluate/understand how preference judgments become proxy rewards and why optimizing those rewards may not capture all human considerations.

Read how Reinforcement Learning from Human Feedback explains preference →

Bayesian Optimization in Action

A user preference is a person's relative desirability judgment over products, experiences, or other alternatives.

How it is used
Frames product recommendation and black-box optimization as learning a user's objective from pairwise choices.
How deep it goes
Defines an inaccessible preference objective and develops one-dimensional setup, comparison encoding, GP modeling, and BayesOpt querying.
Examples
Uses product ratings, production recommendation, and a Hawaiian-shirt search over the number of flowers.
What you can do after
Apply preference optimization by collecting comparisons, encoding outcome indices, and selecting new queries with BayesOpt.

Read how Bayesian Optimization in Action explains preference →

LLM Evaluation and Alignment, The Foundational Ideas

A human preference is a person's comparative judgment that one output or behavior is more desirable than another.

How it is used
Uses comparative judgments to supply a training signal when demonstrations or hand-crafted rewards are difficult.
How deep it goes
Explains the concurrent rollout, labeling, and reward-model loops, including Bradley–Terry scoring and cross-entropy training.
Examples
Illustrates preference learning with simulated robot backflips and Pong, and a harmful trade-off with evasive refusals.
What you can do after
Evaluate preference signals by inspecting labeler agreement, rubric clarity, and unwanted trade-offs.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains preference →

Hidden Influences

A user's inclination toward particular content, items, topics, or qualities of recommendations.

How it is used
Uses preferences to assess personalized recommendations and their role in distributing attention.
How deep it goes
Explains interaction-based learning and discovery trade-offs, then examines why recommendation value is difficult to measure.
Examples
Uses unequal engagement with two accounts, interest in dogs, and an undiscovered interest in books to illustrate preference learning.
What you can do after
Evaluate how interaction-based recommendations can miss interests and why their value is difficult to measure.

Read how Hidden Influences explains preference →

Algorithms Every Programmer Should Know

A preference indicates which potential match a participant favors over another.

How it is used
Models how participants rank potential matches so matching procedures can make and assess assignments.
How deep it goes
Works through ranked lists in a matching example and notes explicit and inferred preferences in applications.
Examples
Uses student–university rankings and dating criteria to illustrate preferences.
What you can do after
Understand how ranked or inferred preferences shape proposals, acceptances, and matching choices.

Read how Algorithms Every Programmer Should Know explains preference →

Deep Learning with Python, Third Edition

The relative ranking of candidate responses by people according to what is more helpful or desirable.

How it is used
Uses human rankings to turn judgments about helpfulness into a training signal for language-model improvement.
How deep it goes
Defines ranking-based data collection and reward modeling, then sketches score-based filtering without deriving the training loss.
Examples
Uses ranked prompt responses and a generate-score-filter-fit loop as concrete illustrations.
What you can do after
Evaluate how human rankings can supervise a reward model and guide response selection.

Read how Deep Learning with Python, Third Edition explains preference →

Also covered in

What to know first

Related concepts

Questions about preference

What is preference?

A preference is a comparative judgment or inclination that favors one alternative, outcome, or characteristic over others.

Which Manning books explain preference?

Reinforcement Learning from Human Feedback, Bayesian Optimization in Action, LLM Evaluation and Alignment, The Foundational Ideas, Hidden Influences, Algorithms Every Programmer Should Know and Deep Learning with Python, Third Edition.

What should I know before learning preference?

Manning books that teach preference usually introduce language model, prompt, fine-tuning and action first.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.