Preference learning problem

As explained in Bayesian Optimization in Action

Preference optimization is optimization of an objective representing a user's preferences when feedback is provided through comparisons rather than numerical evaluations.

A preference learning problem seeks an option a user likes best when feedback comes as comparisons rather than scores. Instead of asking “How would you rate this?”, it asks “Which of these two do you prefer?” This is useful when exact ratings are noisy, difficult to report, or costly to collect, as in A/B testing and product recommendation. section 10.0 section 10.1

Figure 10.1 Examples of user’s preference elicitation in production recommendation. On the left, the user is asked to rate a recommended product. On the right, the user is asked to pick the product they like better. The latter situation helps better elicit the user’s preference.

Comparing two alternatives can be easier for a user than assigning a numerical rating, helping elicit a more reliable preference. section 10.0

The optimization setting is black-box: the underlying preference function is unknown, and each query consumes limited user attention. Pairwise comparisons reduce reporting burden, but each answer reveals only a relative ordering. The task is therefore to learn enough about the user’s latent preference function to select promising alternatives while minimizing intrusive queries. section 10.0 section 10.1

Choosing a Hawaiian shirt

Suppose a shopping site must recommend a shirt with the customer’s preferred number of flowers. Floral-ness forms a one-dimensional search space, from no flowers to many flowers, while the customer’s best level is unknown. Rather than requesting a score for each shirt, the site can show two shirts and record which one wins. A preference learning system uses these outcomes to locate the most preferred region, including distinguishing a local optimum from the overall best choice. section 10.2 section 10.2

A pairwise formulation represents the problem through these pieces:

• Candidate inputs: the alternatives that can be compared, such as shirt designs represented by their floral-ness. section 10.2
• Comparison records: two-element index tensors identifying the winner and loser among observed inputs. section 10.2
• A preference model: a Gaussian process trained on comparisons to make probabilistic predictions about the preference function. section 10.3
• A selection policy: a BayesOpt policy that chooses the next pair for the user to compare. section 10.1

A preference-based Gaussian process learns relative structure rather than an absolute rating scale. If all that is observed is which option wins, shifting or rescaling the hidden objective does not change the comparisons. Predictions can reflect observed winners, and uncertainty decreases around compared inputs. This makes pairwise data sufficient for learning where better options may lie, even when a meaningful numerical scale is unavailable. section 10.3 section 10.3

Figure 10.3 The BayesOpt loop with pairwise comparisons for preference optimization. The GP trains on pairwise comparison data, and the BayesOpt policy decides which pair of data points it should ask the user to compare.

The loop links model learning and decision-making: train a GP on comparison outcomes, then choose another pair that should improve the search. section 10.1

How Bayesian Optimization in Action covers it

How deep it goes

Develops a simulated one-dimensional problem, comparison encoding, and PairwiseGP training, then defers pair-selection decision-making.

Examples

Uses user product ratings, head-to-head car choices, and Hawaiian shirts with varying numbers of flowers.

What it assumes

Introduces preference feedback through practical scenarios before using Python tensors and Gaussian-process model fitting.

What you can do after

Apply it via BoTorch to encode pairwise outcomes and fit a Gaussian-process preference model.

How it connects

Connects pairwise comparisons with Bayesian optimization, Gaussian processes, A/B testing, product recommendation, and multiobjective optimization.

liveBookmanning.com

Read Bayesian Optimization in Action on liveBook Get the book at manning.com

Related concepts

• pairwise comparisonuses · A pairwise comparison evaluates two alternatives against one another to determine their relative preference, value, or rank.
• black-box solverkind of · A black-box solver optimizes an objective through input-output evaluations without requiring access to its internal analytical form or dynamics.
• preference tuningaddressed by · Preference tuning is a model-training technique that adjusts a model’s behavior toward outputs favored by preference signals, judgments, objectives, or rewards.
• Direct Preference Optimizationaddressed by · Direct Preference Optimization is a technique for training a model directly from preference data, such as preferred and dispreferred outputs, without separately fitting a reward model.

This explainer was generated by Manning's knowledge graph from the text of Bayesian Optimization in Action. The section links point to the parts of the book it draws on.