liveBook · Manning

RLHF

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Reinforcement learning from human feedback is a training method that uses human preference judgments as reward information for optimizing a model.

A model can produce fluent responses that are unhelpful, misleading, or unsafe. Reinforcement learning from human feedback (RLHF) uses human preferences to create a training signal that steers the model toward preferred behavior. Rather than writing a correct answer for every case, labelers can compare candidate outputs. section 7.0 section 1.3.2

A common language-model pipeline has three stages:

  • Supervised fine-tuning: Fine-tune a pretrained model on prompts paired with labeler-written demonstrations. section 7.5.1
  • Reward modeling: Sample several responses per prompt, have labelers rank them, and train a reward model to assign scores that predict their preferences. section 7.5.1
  • Reinforcement learning: Sample responses from the fine-tuned model and update it with PPO to favor higher reward scores. A KL penalty keeps the updated policy close to the fine-tuned model, where the reward model is more likely to be reliable. section 7.5.1
Figure 7.8 The InstructGPT three-stage pipeline. Stage 1 fine-tunes the pretrained model on labeler-written demonstrations. Stage 2 trains a reward model on labeler-ranked comparisons of model outputs. Stage 3 optimizes the SFT model with reinforcement learning using PPO, with the reward model's score (penalized by KL divergence from the SFT model) as the reward signal. The SFT model from stage 1 plays two roles: it initializes the RL policy and is the reference distribution in the KL penalty. Source: Figure 2 of Ouyang et al. 2022.

Follow the data and models through the three stages, including the fine-tuned model’s dual role as the starting policy and the KL reference. section 7.5.1

Choosing a financial summary

A labeler compares two earnings summaries. One cites the CFO’s remarks and uses a neutral tone; the other invents a geography and sounds overconfident. Preferring the first supplies a useful training signal, but the single choice blends factuality, instruction-following, and tone. It does not say which difference mattered most. section 7.1.2

A reward score is a learned proxy for preference, not a guarantee of quality. If labelers favor longer answers, a reward model can learn to favor padding. A full-response preference also makes it difficult to identify which tokens or steps earned the label. When a reward model is frozen, policy updates can move into regions where its predictions are less reliable; the KL penalty limits that drift. section 4.3.2 section 7.4.3

For factuality, RLHF requires a reward model capable of assessing factual claims; that evaluator can itself be unreliable. Continue monitoring for hallucinations rather than treating alignment as a replacement for detection. Human labeling also limits scale. Constitutional AI changes the feedback source by using an AI critic guided by principles, while retaining a reward-model and reinforcement-learning pipeline. section 5.4.5 section 8.6

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains the three-stage pipeline, reward modeling, policy updates, and practical limitations including drift and credit assignment.
Examples
Uses competing financial summaries and historical simulated-robot and Atari tasks to illustrate preference-based training.
What it assumes
Introduces reinforcement-learning vocabulary and post-training paradigms before developing the preference-training pipeline.
What you can do after
Evaluate and understand how preference labels guide training and where reward-model signals can fail.
How it connects
Connects preference learning with supervised fine-tuning, reward modeling, PPO, KL divergence, and constitutional AI.

Also explained in

Compare how 18 books explain RLHF →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.