chapter five

5 Reward modeling

 

This chapter covers

  • How language models are trained to predict human preferences
  • How reward models are architected and implemented
  • The different varieties of reward models used today

Reward models are core to the modern approach to RLHF because they are the clever setup that lets us turn the study of human preferences into an optimization problem. They are what enable our models to learn from hard-to-specify signals. They compress complex features in the data into a representation that can be used in downstream training—a sort of magic that once again shows the complex capacity of modern deep learning. These models act as proxy objectives for the core optimization, as studied in the following chapters.

5.1 Training a Bradley–Terry reward model

5.2 Implementation example

5.3 Reward model variants

5.3.1 Preference margin loss

5.3.2 Balancing multiple comparisons per prompt

5.3.3 K-wise loss function

5.4 Outcome reward models

5.5 Process reward models

5.6 Comparing reward model types (and value functions)

5.6.1 Inference across reward model types

5.7 Generative reward modeling (a.k.a. LLM-as-a-judge)

5.8 Further reading

Summary