A reward signal indicates which outcomes are better or worse according to an optimization objective. In RLHF, human evaluators rank candidate responses, a reward model approximates those preferences, and reinforcement learning optimizes the language model to score highly under that learned signal. section 7.1.2
The signal is a proxy for what evaluators value, not a guarantee that the model achieves the underlying intention. When measured approval diverges from intended qualities such as truthfulness, optimizing the signal can encourage sycophancy or reward hacking. section 7.5
An unintended style tendency
A reward signal encouraged a playful “Nerdy” style, and outputs that scored well sometimes included references to goblins and gremlins. The creatures were not directly rewarded, yet the tendency spread to other contexts and was reinforced in later training examples. section 7.1.1
Figure 7.2 Alignment training pipelines. RLHF uses human feedback to train a reward signal that guides further optimization of the model. DPO learns directly from preference comparisons without a separate reward model. Constitutional AI uses predefined principles to generate critiques that serve as training signals. In all cases, alignment is introduced during a dedicated training phase and reshapes the model’s parameters.

Compare how RLHF uses a reward signal with DPO’s direct use of preference comparisons and Constitutional AI’s principle-based training signals. section 7.1.2















