liveBook · Manning

Impossibility theorem

As explained in Reinforcement Learning from Human Feedback

An impossibility theorem proves that no rule can simultaneously satisfy a specified set of desirable conditions.

An impossibility theorem proves that no preference-aggregation rule can satisfy every specified fairness criterion at once. In social choice theory, this limits what optimization methods can guarantee when combining individual preferences. section 10.5 section 10.6

The result matters for RLHF because human preferences are noisy, context-dependent, temporally shifting, and sometimes non-transitive. Aggregation therefore requires trade-offs rather than a universally fair solution. section 10.5 section 10.6

Practical implications include:

  • Treat scalar utility models as useful abstractions rather than complete representations of human values. section 10.5 section 10.6
  • Expect tensions among fairness, value alignment, and corrigibility when preference models guide AI behavior. section 10.5
  • Focus RLHF on tractable objectives such as style and performance instead of claiming to resolve human values completely. section 10.6

Preference aggregation in RLHF

When preferences differ across people, places, or situations, one aggregation method cannot preserve every fairness criterion simultaneously. An RLHF pipeline can still optimize collected judgments for style or performance, but its result reflects chosen trade-offs and modeling assumptions. section 10.5 section 10.6

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Defines the simultaneous-criteria limit and applies it to RLHF, but does not provide a formal theorem statement or proof.
Examples
Uses preference conflicts across people, places, and situations, plus RLHF datasets optimized for style and performance.
What it assumes
Assumes preference modeling and aggregation concepts while highlighting that human preferences violate stability, comparability, and transitivity assumptions.
What you can do after
Evaluate preference-aggregation methods by identifying unavoidable fairness trade-offs and the practical scope of RLHF objectives.
How it connects
Connects social choice theory with fairness criteria, preference aggregation, utility functions, RLHF, value alignment, and corrigibility.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.