liveBook · Manning

Reward signal

As explained in Machines that Think

A reward signal is feedback indicating how desirable a system's behavior or output is according to an optimization objective.

A reward signal indicates which outcomes are better or worse according to an optimization objective. In RLHF, human evaluators rank candidate responses, a reward model approximates those preferences, and reinforcement learning optimizes the language model to score highly under that learned signal. section 7.1.2

The signal is a proxy for what evaluators value, not a guarantee that the model achieves the underlying intention. When measured approval diverges from intended qualities such as truthfulness, optimizing the signal can encourage sycophancy or reward hacking. section 7.5

An unintended style tendency

A reward signal encouraged a playful “Nerdy” style, and outputs that scored well sometimes included references to goblins and gremlins. The creatures were not directly rewarded, yet the tendency spread to other contexts and was reinforced in later training examples. section 7.1.1

Figure 7.2 Alignment training pipelines. RLHF uses human feedback to train a reward signal that guides further optimization of the model. DPO learns directly from preference comparisons without a separate reward model. Constitutional AI uses predefined principles to generate critiques that serve as training signals. In all cases, alignment is introduced during a dedicated training phase and reshapes the model’s parameters.

Compare how RLHF uses a reward signal with DPO’s direct use of preference comparisons and Constitutional AI’s principle-based training signals. section 7.1.2

How Machines that Think covers it

How deep it goes
Explains reward-model training and subsequent optimization, then notes that measured approval can diverge from intended behavior.
Examples
Uses ranked candidate responses and a playful-style training quirk involving goblins and gremlins.
What you can do after
Understand how learned feedback guides optimization and why optimizing a proxy can produce unintended behaviors.
How it connects
Links reward signals to RLHF, reward models, human preference comparisons, and the contrast with DPO.

Also explained in

Compare how 16 books explain reward signal →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.