Why does one answer feel more helpful than another? A reward model is a separate learned system that scores responses according to preferences inferred from human feedback. It helps an LLM learn that language quality means more than predicting likely text: it also means producing answers people find useful, clear, accurate, safe, and appropriately toned. section 2.2.3
In a typical reinforcement learning from human feedback (RLHF) loop:
- The LLM generates several responses to the same prompt. section 2.2.3
- Human reviewers rank those responses by criteria such as helpfulness, clarity, accuracy, safety, and tone. section 2.2.3
- The rankings train the reward model to score responses in ways that approximate those preferences. section 2.2.3
- The reward signal guides updates to the LLM, making highly preferred responses more likely and disliked responses less likely. section 2.2.3
A preference the model can learn
Suppose reviewers compare a concise explanation with a confusing one, or a polite refusal with an answer that gives dangerous instructions. Repeated rankings teach the reward model to score the clearer or safer response more highly. Over time, these learned preferences contribute to guardrails: behaviors that shape what the assistant tends to do without relying on a hand-written list of forbidden topics. section 2.2.3
Figure 2.8 The RLHF loop applied to a single prompt. The model generates two candidate responses; human reviewers rank them by helpfulness and clarity; a reward model learns that preference; the LLM is updated to make better responses more likely. The process repeats until the model's behavior aligns with human expectations.

Follow the RLHF loop from candidate responses through human ranking and reward-model learning to updated LLM behavior. section 2.2.3









