10 The nature of preferences
This chapter covers
- The complex, interdisciplinary nature of human preferences
- Why RLHF will never be a fully solved problem
- The assumptions baked into RLHF’s problem setup
Reinforcement learning from human feedback, also referred to as reinforcement learning from human preferences in early literature, emerged to optimize machine learning models in domains where specifically designing a reward function is hard. The word preferences is at the center of the RLHF process: human preferences are what we’re trying to model and what fuels the data for training. To understand the scope of the challenge in modeling and measuring human preferences, a broader context is needed in understanding what a preference is, how our understanding of preferences emerged, and how multiple fields of economics, psychology, philosophy, and more inform modern RLHF.
10.1 When preference replaces correctness
Consider an example. On February 26, 2025, I asked both Claude 3.7 Sonnet and ChatGPT with GPT-4o to “Write me a short poem about an optimistic goldfish.” How do you decide which of these two poems is better?
Example 1: