A reward signal is a single numerical score used to guide optimization. For a language model, a reward function can score a prompt and response; higher scores indicate responses the training process should favor. The score may come from rules, a human, or an AI model. section 7.3.1
An evaluation score measures how good an output is; a reward uses that kind of score to change future behavior. Measurement alone does not fix a failure: an optimizer must use the signal to update the model or system. section 7.1 section 1.3.1
Figure 7.4 The reinforcement learning training loop. A prompt enters the policy, which samples a response. The response is scored by a reward function. The scalar reward drives a gradient update that adjusts the policy's parameters. The loop repeats over many prompts and sampled responses
The loop connects a sampled response, its scalar reward, and the policy update. section 7.3.1
Choose a reward according to what can be checked:
- For a verifiable task, use a programmatic check where possible. A math answer, passing unit tests, or a required output format can receive 1 for success and 0 otherwise. section 7.1.1
- For an open-ended task, such as dialogue or summarization, multiple answers may be valid. A reward model trained on human preferences can help capture judgments that are difficult to specify as fixed rules. section 7.1.1 section 7.4.2
Scoring a financial summary
A model drafts a quarterly earnings summary. A reward function scores its faithfulness, citations, and utility; an optimizer then makes high-scoring responses more likely. A single score, however, does not reveal whether accurate figures, citations, or writing style caused it. section 7.3.1 section 7.3.2
A reward is a proxy for the desired outcome, not a guarantee of it. Optimization can exploit gaps in the score: rewarding engagement alone may encourage clickbait, while rewarding safety alone may encourage blanket refusals. This is reward hacking, a form of Goodhart’s law. Check whether higher scores correspond to the behavior users actually need. section 1.3.1 section 7.8















