How deeply does Reinforcement Learning from Human Feedback cover proxy reward function?

Defines reward-model scoring and explains over-optimization, but stops short of detailed RLHF optimization choices.

Proxy reward function

As explained in Reinforcement Learning from Human Feedback

A proxy reward is a reward signal that approximates a desired human or task objective rather than measuring it directly.

A proxy reward function supplies an optimization signal that approximates a desired human or task objective instead of measuring it directly. In RLHF, the proxy reward is the score produced by a trained reward model for the reinforcement-learning algorithm. Because that score is only correlated with downstream usefulness, increasing it indefinitely can reduce actual model quality. section 14.1.1 section 3.0

Use the concept to distinguish optimization progress from genuine task success:

Treat the reward-model score as an approximation of user preferences, not as a complete measure of downstream satisfaction. section 14.1.1
Combine proxy-reward optimization with a distance-based regularizer in the RLHF objective. section 3.0
Monitor for over-optimization: training reward rises while downstream quality first improves, then declines. section 14.1.1
Separate over-optimization from overfitting: the former reflects a flawed metric, while the latter reflects poor generalization across data. section 14.1.1

RLHF training reward versus usefulness

A language model receives increasingly high scores from its reward model during RL training. Downstream quality improves initially, reaches a peak, and then falls as optimization continues. The model has improved at the proxy objective while moving away from actual user satisfaction. section 14.1.1

How Reinforcement Learning from Human Feedback covers it

Cover of Reinforcement Learning from Human Feedback

liveBook manning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Related concepts

reward modelproduced by · A reward model is a learned model that assigns scores to outputs, actions, or outcomes to represent their desirability according to preferences or evaluation criteria.
reinforcement learning with human feedbackused by · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.