Preference modeling

As explained in Bayesian Optimization in Action

Preference learning is the process of learning a user's preferences from judgments or comparisons rather than necessarily from numerical evaluations.

Preference modeling learns what a user prefers from judgments or pairwise comparisons rather than requiring numerical evaluations. In Bayesian optimization, the system presents two candidate points and uses the user's choice to guide the search. section 10.4

A practical “king of the hill” strategy pairs the best point found so far—the incumbent—with a challenger selected by a BayesOpt acquisition policy. Repeating this comparison balances exploitation of promising regions with exploration of uncertain ones. section 10.4

Choosing the next comparison

Use an acquisition policy such as UpperConfidenceBound or qNoisyExpectedImprovement to optimize an acquisition function, stack its challenger with the incumbent, collect the user's comparison, and append the resulting preference data.

Figure 10.7 Illustration of the “king of the hill” strategy in Bayesian preference optimization. We compare the best point we have seen so far against a promising candidate identified by a BayesOpt policy.

Illustrates the incumbent-versus-challenger comparison at the center of Bayesian preference optimization. section 10.4

How Bayesian Optimization in Action covers it

How deep it goes

Develops a reusable incumbent–challenger procedure and implementation pattern, while deferring preference-model internals.

Examples

Uses king-of-the-hill comparisons, incumbent and challenger points, UpperConfidenceBound, and qNoisyExpectedImprovement.

What it assumes

Assumes familiarity with Bayesian optimization policies and acquisition scores, while treating pairwise preference collection as the new setting.

What you can do after

Apply Bayesian optimization policies to construct pairwise comparisons and iteratively collect preference data.

How it connects

Connects preference learning with Bayesian optimization, acquisition functions, and exploration–exploitation tradeoffs.

liveBookmanning.com

Read Bayesian Optimization in Action on liveBook Get the book at manning.com

Related concepts

• human-in-the-loopuses · Human-in-the-loop is a system pattern in which people provide input, judgment, review, or intervention within an automated process.
• reward signaluses · A reward signal is an evaluative feedback value representing the desirability or payoff of an action, outcome, or state.
• direct alignment algorithmaddressed by · A direct alignment algorithm optimizes a model directly from preference data without learning an intermediate reward model or using a reinforcement-learning optimizer.
• alignment trainingused by · Alignment training is the process of adjusting an AI system’s behavior to better conform to intended human goals, preferences, values, or constraints.
• pairwise comparisonuses · A pairwise comparison evaluates two alternatives against one another to determine their relative preference, value, or rank.
• objective functioncontrasts with · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.

This explainer was generated by Manning's knowledge graph from the text of Bayesian Optimization in Action. The section links point to the parts of the book it draws on.