A model can produce fluent responses that are unhelpful, misleading, or unsafe. Reinforcement learning from human feedback (RLHF) uses human preferences to create a training signal that steers the model toward preferred behavior. Rather than writing a correct answer for every case, labelers can compare candidate outputs. section 7.0 section 1.3.2
A common language-model pipeline has three stages:
- Supervised fine-tuning: Fine-tune a pretrained model on prompts paired with labeler-written demonstrations. section 7.5.1
- Reward modeling: Sample several responses per prompt, have labelers rank them, and train a reward model to assign scores that predict their preferences. section 7.5.1
- Reinforcement learning: Sample responses from the fine-tuned model and update it with PPO to favor higher reward scores. A KL penalty keeps the updated policy close to the fine-tuned model, where the reward model is more likely to be reliable. section 7.5.1
Figure 7.8 The InstructGPT three-stage pipeline. Stage 1 fine-tunes the pretrained model on labeler-written demonstrations. Stage 2 trains a reward model on labeler-ranked comparisons of model outputs. Stage 3 optimizes the SFT model with reinforcement learning using PPO, with the reward model's score (penalized by KL divergence from the SFT model) as the reward signal. The SFT model from stage 1 plays two roles: it initializes the RL policy and is the reference distribution in the KL penalty. Source: Figure 2 of Ouyang et al. 2022.
Follow the data and models through the three stages, including the fine-tuned model’s dual role as the starting policy and the KL reference. section 7.5.1
Choosing a financial summary
A labeler compares two earnings summaries. One cites the CFO’s remarks and uses a neutral tone; the other invents a geography and sounds overconfident. Preferring the first supplies a useful training signal, but the single choice blends factuality, instruction-following, and tone. It does not say which difference mattered most. section 7.1.2
A reward score is a learned proxy for preference, not a guarantee of quality. If labelers favor longer answers, a reward model can learn to favor padding. A full-response preference also makes it difficult to identify which tokens or steps earned the label. When a reward model is frozen, policy updates can move into regions where its predictions are less reliable; the KL penalty limits that drift. section 4.3.2 section 7.4.3
For factuality, RLHF requires a reward model capable of assessing factual claims; that evaluator can itself be unreliable. Continue monitoring for hallucinations rather than treating alignment as a replacement for detection. Human labeling also limits scale. Constitutional AI changes the feedback source by using an AI critic guided by principles, while retaining a reward-model and reinforcement-learning pipeline. section 5.4.5 section 8.6

















