liveBook · Manning

Reward signal

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A scalar signal is a single numerical value that summarizes an evaluation or learning outcome.

A reward signal is a single numerical score used to guide optimization. For a language model, a reward function can score a prompt and response; higher scores indicate responses the training process should favor. The score may come from rules, a human, or an AI model. section 7.3.1

An evaluation score measures how good an output is; a reward uses that kind of score to change future behavior. Measurement alone does not fix a failure: an optimizer must use the signal to update the model or system. section 7.1 section 1.3.1

Figure 7.4 The reinforcement learning training loop. A prompt enters the policy, which samples a response. The response is scored by a reward function. The scalar reward drives a gradient update that adjusts the policy's parameters. The loop repeats over many prompts and sampled responses

The loop connects a sampled response, its scalar reward, and the policy update. section 7.3.1

Choose a reward according to what can be checked:

  • For a verifiable task, use a programmatic check where possible. A math answer, passing unit tests, or a required output format can receive 1 for success and 0 otherwise. section 7.1.1
  • For an open-ended task, such as dialogue or summarization, multiple answers may be valid. A reward model trained on human preferences can help capture judgments that are difficult to specify as fixed rules. section 7.1.1 section 7.4.2

Scoring a financial summary

A model drafts a quarterly earnings summary. A reward function scores its faithfulness, citations, and utility; an optimizer then makes high-scoring responses more likely. A single score, however, does not reveal whether accurate figures, citations, or writing style caused it. section 7.3.1 section 7.3.2

A reward is a proxy for the desired outcome, not a guarantee of it. Optimization can exploit gaps in the score: rewarding engagement alone may encourage clickbait, while rewarding safety alone may encourage blanket refusals. This is reward hacking, a form of Goodhart’s law. Check whether higher scores correspond to the behavior users actually need. section 1.3.1 section 7.8

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains the scoring and policy-update loop, reward sources, preference training, and limitations including credit assignment and reward hacking.
Examples
Illustrates rewards with binary math and code checks, preference comparisons, and scores for financial earnings summaries.
What you can do after
Evaluate and understand how reward choices steer training and where a score may diverge from the intended outcome.
How it connects
Connects reward signals to reinforcement learning, evaluation metrics, human preferences, programmatic verification, and Goodhart’s law.

Also explained in

Compare how 16 books explain reward signal →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.