Preference aggregation combines judgments across people, places, and situations into representations that preference models can use. In RLHF, it addresses conflicts created by changing preferences, differing contexts, and multiple principals rather than assuming one stable human value function. section 10.5
Aggregation becomes difficult when preference data violate assumptions used by standard utility models:
- Measured preferences may be non-transitive or incomparable as the environment becomes more complex. section 10.5
- Implicit signals such as page-view time, closing a tab, or repeating a question may interact with later training and deployment. section 10.5
- Results can change when respondents choose among more than two options or provide judgments at different times and in different contexts. section 10.5
- Aggregated preference data can conceal contextual differences between users when respondent accuracy is low. section 10.5
A changing preference signal
A preference pipeline might treat repeated questions to a language model as evidence of dissatisfaction, while the same behavior could reflect a different context or a changing preference. Aggregation therefore risks optimizing a proxy without resolving why judgments differ. section 10.5
No single aggregation method can satisfy every fairness criterion simultaneously. The practical goal is therefore narrower: use preference data for tractable style and performance objectives while recognizing that human values remain noisy, context-dependent, temporally shifting, and sometimes non-transitive. section 10.6
