A preference is a person's relative desirability judgment over products, experiences, or other alternatives. In recommendation and A/B testing, numerical ratings can be noisy because mood, advertising, delivery, or other external factors distort the customer's true evaluation. Asking which of two alternatives is better can reduce the reporting burden, though it provides less information than a numerical score. section 10.0 section 10.5
Preference optimization turns pairwise judgments into an iterative black-box search:
- Represent the unknown true preference as an objective over products or experiences. section 10.1
- Train a Gaussian process on comparisons rather than numerical evaluations, preserving higher predictions at preferred locations and uncertainty elsewhere. section 10.1 section 10.5
- Use a Bayesian optimization policy to choose a new pair, balancing exploitation of promising regions with exploration of uncertain ones. section 10.1
- Record the preferred and nonpreferred item as two indices, then append the result to the training data. section 10.2 section 10.5
Choosing a Hawaiian shirt
Suppose a shopper mainly cares about how many flowers appear on a Hawaiian shirt. The search space runs from no flowers to shirts covered in flowers, while the shopper's preference curve remains hidden. Pairwise comparisons can locate a shirt with nearly the maximum preference, even when the curve has both a local optimum and a better global optimum near the upper bound. section 10.2
