Chapter 2: A tiny history of RLHF
This introduction explains why RLHF’s recent history matters for understanding its present-day pipeline.
After this chapter you will be able to
- RLHF’s recent historical roots. Explain that RLHF was formalized recently, that much of its documentation remains in academic papers, and that work on applications unlike modern language models helped shape today’s pipeline.
- Trajectory preference learning signal. Explain that Christiano et al. (2017) applied RLHF to human preferences between agents’ Atari trajectories, and that choosing between trajectories could be a more effective learning signal in some domains than direct environment interaction.
- Early RLHF method continuity. Explain why the 2019 language-model work is recognizable as RLHF today, despite changes in evaluation tasks and model capabilities.
- ChatGPT RLHF lineage and adoption. Explain that ChatGPT used RLHF methods similar to InstructGPT’s, with slight differences in data collection, and that RLHF was subsequently used in other leading language models.
- Expansion into preference fine-tuning. Describe how the section frames RLHF as growing into the broader field of preference fine-tuning, naming its examples: process rewards for reasoning steps, direct alignment algorithms, and feedback from code or math execution and other online reasoning methods.
What it builds on
- RLHF in three steps (chapter 1) · Assumes the introductory account of human feedback in RLHF.
Section by section
Introduction
This introduction explains why RLHF’s recent history matters for understanding its present-day pipeline.
- RLHF’s recent historical roots: Explain that RLHF was formalized recently, that much of its documentation remains in academic papers, and that work on applications unlike modern language models helped shape today’s pipeline.
Why it matters: This context helps explain why the pipeline has its current shape rather than treating it as a design invented solely for language models.
Key terms: RLHF pipeline, academic literature, seminal work
2.1 Origins to 2018: RL on preferences
This section traces early reinforcement learning from human feedback, from action-level evaluations to trajectory preferences and the later use of reward models in alignment research.
- Early action-level feedback methods: Distinguish how TAMER and COACH used human feedback: TAMER used repeated scores of an agent’s actions to train a reward model for learning a policy, while COACH used positive and negative feedback to tune an advantage function.
Why it matters: These examples show that human feedback shaped RL before modern language-model RLHF. - Trajectory preference learning signal: Explain that Christiano et al. (2017) applied RLHF to human preferences between agents’ Atari trajectories, and that choosing between trajectories could be a more effective learning signal in some domains than direct environment interaction.
Why it matters: Trajectory comparisons are a central historical example of learning from human preferences. - Reward models beyond RL tasks: Describe the shift at the end of this era from treating reward models as tools for solving RL problems to proposing them as a way to study alignment.
Why it matters: This shift connects the early preference-learning work to the book’s broader alignment focus.
Key terms: TAMER, COACH, reward model, advantage function, Christiano et al. (2017), preferences, trajectories, learning signal, reward models, alignment
2.2 2019 to 2022: RL from human preferences on language models
This section traces how early RLHF methods for language models persisted as applications expanded and new research concerns emerged.
- Early RLHF method continuity: Explain why the 2019 language-model work is recognizable as RLHF today, despite changes in evaluation tasks and model capabilities.
Why it matters: It connects the origins of language-model RLHF to the methods discussed later in the book. - Expanding RLHF applications: Describe how RLHF spread across language-model tasks and into work on chat models during this period.
Why it matters: The range of applications shows that RLHF was not confined to a single language-model task. - Emerging RLHF research concerns: Identify the research areas the section highlights beyond applications, including what it says over-optimization and red teaming mean.
Why it matters: These concerns show that RLHF research also examined limitations and safety, not just new uses.
Key terms: reinforcement learning from human preferences, reward models, KL divergence, feedback pipeline, summarization, instruction following, question-answering, general dialogue, chat models, reward model over-optimization
2.3 2023 to the present: The ChatGPT era
The section traces RLHF from ChatGPT’s training to its widespread use and its expansion into preference fine-tuning.
- ChatGPT RLHF lineage and adoption: Explain that ChatGPT used RLHF methods similar to InstructGPT’s, with slight differences in data collection, and that RLHF was subsequently used in other leading language models.
Why it matters: This places ChatGPT in the history of RLHF and shows that the approach spread beyond one model. - Expansion into preference fine-tuning: Describe how the section frames RLHF as growing into the broader field of preference fine-tuning, naming its examples: process rewards for reasoning steps, direct alignment algorithms, and feedback from code or math execution and other online reasoning methods.
Why it matters: This broader framing connects RLHF’s history to the range of training approaches discussed later.
Key terms: ChatGPT, RLHF, InstructGPT, preference fine-tuning (PreFT), process rewards, direct alignment algorithms, direct preference optimization (DPO), execution feedback
2.4 Summary
Where this is used later
- Training a Bradley–Terry reward model (chapter 5) · Provides historical context for training reward models from human feedback.
- Why we need preference data (chapter 11) · Provides an early example of why preference data can serve as a learning signal.
- The origins of RLHF and preferences (chapter 10) · Supplies historical context for the origins of RLHF and preferences.
- KL divergence in RL optimization (chapter 15) · Develops the use of KL divergence in RL optimization.
- Quantitative over-optimization (chapter 14) · Examines quantitative over-optimization in more detail.
- InstructGPT (chapter 3) · The InstructGPT recipe provides more detail on the earlier approach named here.
- Process reward models (chapter 5) · Develops the process-reward direction named here.
- Direct preference optimization (chapter 8) · Develops direct preference optimization, named here as an inspiration for direct alignment algorithms.
Concepts explained in this chapter
- reinforcement learning with human feedbackReinforcement learning from human feedback (RLHF) is a model-alignment technique that optimizes behavior using human preference signals, often through a learned reward model or direct preference objective.
- reward modelA reward model is a learned model that predicts the reward or preference value assigned to an output or behavior.
- Kullback–Leibler divergenceForward KL divergence is KL divergence from a target distribution to an approximating distribution, with expectation taken under the target distribution.
- preferenceA human preference is a person's comparative judgment about the desirability or quality of model behavior or outputs.
- preference tuningPreference optimization is model training that adjusts behavior toward preferred outputs using preference-based objectives or rewards.
- alignmentAlignment is the process of shaping a model's behavior and outputs to conform to desired human or task-related objectives.
- over-optimizationReward over-optimization is a condition in which a reward model's score continues to improve while actual quality measured by held-out evaluations or human judgment degrades.
- direct alignment algorithmA direct alignment algorithm optimizes a model directly from preference data without learning an intermediate reward model.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.