The Von Neumann–Morgenstern utility theorem supplies conditions under which preferences over uncertain outcomes can be represented by a scalar utility function and optimized through expected value. In reinforcement learning, this gives a theoretical basis for converting preference judgments into numerical objectives rather than treating every choice as irreducibly qualitative. section 10.5 section 10.6
Applied to RLHF and related preference-optimization systems, the theorem has both engineering value and important limits:
- An MDP can theoretically be modified to accommodate VNM-style utility, but this formulation is rarely used in practice. section 10.5
- Markovian assumptions restrict expressivity, while partially observed processes make precise specification especially difficult for language tasks. section 10.5
- The representation assumes properties such as transitivity, comparability, and stability that human preferences often violate through noise, context dependence, temporal change, or intransitivity. section 10.6
- Numerical preference models can omit contextual influences such as visual choice presentation, and choice behavior does not automatically establish that one option is preferred. section 10.5
- Social-choice impossibility results show that one aggregation method cannot satisfy every fairness criterion simultaneously, complicating multi-person or multi-principal alignment. section 10.5
Preference modeling in RLHF
A preference-optimization pipeline may assign scalar scores to alternative model outputs and optimize expected value. The theorem legitimizes this reduction, while the surrounding evidence warns that scores can change with presentation, context, time, or inconsistent human judgments. section 10.5 section 10.6
