liveBook · Manning

Reward model

As explained in Understanding AI

A reward model is a learned model that scores outputs according to preferences inferred from human or other evaluative feedback.

Why does one answer feel more helpful than another? A reward model is a separate learned system that scores responses according to preferences inferred from human feedback. It helps an LLM learn that language quality means more than predicting likely text: it also means producing answers people find useful, clear, accurate, safe, and appropriately toned. section 2.2.3

In a typical reinforcement learning from human feedback (RLHF) loop:

  • The LLM generates several responses to the same prompt. section 2.2.3
  • Human reviewers rank those responses by criteria such as helpfulness, clarity, accuracy, safety, and tone. section 2.2.3
  • The rankings train the reward model to score responses in ways that approximate those preferences. section 2.2.3
  • The reward signal guides updates to the LLM, making highly preferred responses more likely and disliked responses less likely. section 2.2.3

A preference the model can learn

Suppose reviewers compare a concise explanation with a confusing one, or a polite refusal with an answer that gives dangerous instructions. Repeated rankings teach the reward model to score the clearer or safer response more highly. Over time, these learned preferences contribute to guardrails: behaviors that shape what the assistant tends to do without relying on a hand-written list of forbidden topics. section 2.2.3

Figure 2.8 The RLHF loop applied to a single prompt. The model generates two candidate responses; human reviewers rank them by helpfulness and clarity; a reward model learns that preference; the LLM is updated to make better responses more likely. The process repeats until the model's behavior aligns with human expectations.

Follow the RLHF loop from candidate responses through human ranking and reward-model learning to updated LLM behavior. section 2.2.3

How Understanding AI covers it

How deep it goes
Defines the scoring role and traces rankings through reward-model learning to LLM updates, but stops short of implementation details.
Examples
Contrasts concise explanations, confusing answers, polite refusals, and dangerous instructions through reviewer rankings.
What it assumes
Assumes the distinction between pretraining and fine-tuning, while introducing preference scoring through concrete human rankings.
What you can do after
Evaluate how human rankings become learned preferences that guide an LLM toward more helpful, safe, and trustworthy responses.
How it connects
Connects reinforcement learning from human feedback, alignment training, fine-tuning, human preferences, and guardrails.

Also explained in

Compare how 10 books explain reward model →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Understanding AI. The section links point to the parts of the book it draws on.