Chapter 1: Introduction
The introduction explains why RLHF arose and briefly traces its move from traditional reinforcement-learning applications to prominence with ChatGPT.
After this chapter you will be able to
- Preference signals for difficult objectives. Explain why RLHF uses human preference signals to guide optimization when the desired behavior of a human-facing AI system is hard to specify.
- Three-stage RLHF pipeline. Trace the basic RLHF pipeline from training an instruction-following language model, through collecting human preference data and training a reward model, to optimizing sampled generations using that model's ratings.
- Post-training method roles. Distinguish the three optimization methods in post-training by what each is used to improve, and place RLHF within preference fine-tuning.
- Style and format as value. Distinguish style or tone from answer format, and explain why training for qualities such as warmth and engagement is more than a superficial change.
- Response-level feedback versus token prediction. Explain the section’s contrast between IFT’s per-token training and RLHF’s response-level feedback, including the role of better and worse responses.
- RLHF flexibility and limitations. Explain why RLHF’s flexibility does not make it a stand-alone solution, citing the section’s concerns about reward proxies, optimization control, and cost.
- Instruction-following model foundation. Explain how the first stage turns a base text-completion model into an instruction-following, question-answering model.
- Human preference reward signal. Describe how the second stage produces a reward signal for judging the quality of model answers.
- Reward-guided model updates. Trace how the final stage uses the question-answering model and reward model to improve generated answers.
- Elicitation of base-model potential. Explain why post-training can substantially improve a model without changing most of its pretraining: the base model contains broad potential, while post-training makes more of it accessible in useful responses.
- Limits of superficial alignment. Distinguish the claim that alignment mainly teaches conversational style from the section’s argument that post-training can also substantially change behavior and performance, including reasoning.
- Early open post-training. Describe how early open models sought ChatGPT-like behavior and why open groups were slow to adopt RLHF.
- DPO preference-tuning shift. Explain why DPO marked a shift in open post-training and how preference tuning became an expectation for good model releases.
- Multistage post-training landscape. Distinguish basic open preference-tuning recipes from the fuller post-training processes described here, and state where the section places newer innovation.
- RLHF as a values lens. Explain why the section treats RLHF as more than a way to improve performance on tasks.
Section by section
Introduction
The introduction explains why RLHF arose and briefly traces its move from traditional reinforcement-learning applications to prominence with ChatGPT.
- Preference signals for difficult objectives: Explain why RLHF uses human preference signals to guide optimization when the desired behavior of a human-facing AI system is hard to specify.
Why it matters: This motivation explains why human preferences are central to RLHF. - Early applications and later prominence: Distinguish RLHF's early applications in traditional reinforcement-learning domains from the later developments that made it widely known.
Why it matters: RLHF's history shows that it did not begin with chatbots, even though chatbots brought it wider attention.
Key terms: reinforcement learning from human feedback, human information, preference signals, optimization, control problems, reinforcement learning, ChatGPT, large language models, foundation models
1.1 RLHF in three steps
This section outlines the three-step RLHF pipeline and places it within the broader, evolving practice of language-model post-training.
- Three-stage RLHF pipeline: Trace the basic RLHF pipeline from training an instruction-following language model, through collecting human preference data and training a reward model, to optimizing sampled generations using that model's ratings.
Why it matters: This sequence gives you a framework for understanding the training stages discussed throughout the book. - Post-training method roles: Distinguish the three optimization methods in post-training by what each is used to improve, and place RLHF within preference fine-tuning.
Why it matters: Separating these roles helps explain what RLHF contributes without treating it as all of post-training. - RLHF prominence and complexity: Explain why RLHF drew early attention in post-training, why it is now described as one component among others, and what makes preference fine-tuning more complex than instruction fine-tuning.
Why it matters: This explains both the book's emphasis on RLHF and why it discusses complementary training methods.
Key terms: RLHF, human preference data, reward model, RL optimizer, post-training, instruction fine-tuning, supervised fine-tuning, preference fine-tuning, reinforcement learning with verifiable rewards, proxy reward models
1.2 What does RLHF do?
This section explains how RLHF changes language-model responses, why those changes matter, how its training signal differs from instruction fine-tuning, and what limits its use.
- Pretraining and post-training responses: Describe how the base model and the post-trained model answer the same question differently: one continues text and adds unrelated material, while the other gives a concise answer.
Why it matters: The contrast makes the practical effect of post-training visible before the section explains its methods. - Style and format as value: Distinguish style or tone from answer format, and explain why training for qualities such as warmth and engagement is more than a superficial change.
Why it matters: A useful answer depends not just on its information, but also on how it is conveyed. - Response-level feedback versus token prediction: Explain the section’s contrast between IFT’s per-token training and RLHF’s response-level feedback, including the role of better and worse responses.
Why it matters: This difference explains why feedback can shape broader response qualities without prescribing one exact answer. - RLHF flexibility and limitations: Explain why RLHF’s flexibility does not make it a stand-alone solution, citing the section’s concerns about reward proxies, optimization control, and cost.
Why it matters: The benefits of RLHF have to be weighed against the difficulty of optimizing an imperfect reward signal.
Key terms: autoregressive pretraining, post-training, style, tone, format, IFT, RLHF, per-token update, response level, negative feedback
1.3 Walkthrough of an RLHF recipe
This section walks through the three stages of a minimal RLHF recipe: instruction fine-tuning, reward modeling, and reinforcement learning.
- Instruction-following model foundation: Explain how the first stage turns a base text-completion model into an instruction-following, question-answering model.
Why it matters: This stage gives the model the response format that the later stages aim to improve. - Human preference reward signal: Describe how the second stage produces a reward signal for judging the quality of model answers.
Why it matters: RL needs a quality signal, and the reward model supplies one based on human preferences. - Reward-guided model updates: Trace how the final stage uses the question-answering model and reward model to improve generated answers.
Why it matters: This connects preference-based scoring to changes in what the final assistant is likely to generate.
Key terms: base model, instruction-following model, next-token prediction loss function, assistant persona, reward model, human preferences, preference relations, scalar signal, prompts, completions
1.4 An intuition for post-training
This section explains post-training as a way to draw out a base model’s capabilities, argues that its effects go beyond conversational style, and notes its growing computational scale.
- Elicitation of base-model potential: Explain why post-training can substantially improve a model without changing most of its pretraining: the base model contains broad potential, while post-training makes more of it accessible in useful responses.
Why it matters: It explains how substantial gains can come from working with an existing base model. - Limits of superficial alignment: Distinguish the claim that alignment mainly teaches conversational style from the section’s argument that post-training can also substantially change behavior and performance, including reasoning.
Why it matters: It prevents narrow style improvements from being mistaken for the full extent of post-training’s effects. - Growing post-training compute: Describe the section’s observation that reinforcement-learning post-training is using more compute, while its methods for scaling remain at an early stage.
Why it matters: It places the section’s argument about post-training’s importance in the context of how frontier models are being trained.
Key terms: Elicitation Theory of Post-training, base model, post-training, next-token prediction, Superficial Alignment Hypothesis, alignment, style, IFT, RLVR, RL
1.5 How we got here
This section traces how post-training moved from early open-model fine-tuning and skepticism about RLHF through DPO-based preference tuning toward more complex, multistage recipes and newer reasoning methods.
- Early open post-training: Describe how early open models sought ChatGPT-like behavior and why open groups were slow to adopt RLHF.
Why it matters: This explains why impressive early open models did not immediately close the post-training gap. - DPO preference-tuning shift: Explain why DPO marked a shift in open post-training and how preference tuning became an expectation for good model releases.
Why it matters: It shows how a simpler preference-based approach changed what open post-training recipes were expected to include. - Multistage post-training landscape: Distinguish basic open preference-tuning recipes from the fuller post-training processes described here, and state where the section places newer innovation.
Why it matters: It puts individual tuning methods in the context of a broader training process.
Key terms: post-training, self-instruct, instruction tuning, RLHF, direct preference optimization, DPO, pairwise preference data, preference-tuning, full post-training, IFT
1.6 Future of RLHF
The section frames RLHF both as a bridge to broader use of reinforcement learning in language-model fine-tuning and as a way to study how AI systems can reflect human values and objectives.
- RLHF as a values lens: Explain why the section treats RLHF as more than a way to improve performance on tasks.
Why it matters: This broader question explains why RLHF remains important beyond improvements on particular tasks. - Bridge to broader RL investment: Describe why the section calls RLHF a bridge to further investment in RL for fine-tuning large base models.
Why it matters: It explains why interest in the RL part of RLHF may grow without displacing the section’s broader view of RLHF.
Key terms: RLHF, human values and objectives, RL methods, fine-tuning large base models, o1 reasoning models
1.7 Summary
Where this is used later
- When preference replaces correctness (chapter 10) · Develops the distinction between preference and correctness.
- Why we need preference data (chapter 11) · Examines why preference data is needed.
- Canonical training recipes (chapter 3) · Canonical training recipes build on this outline of the stages.
- Origins to 2018: RL on preferences (chapter 2) · Assumes the introductory account of human feedback in RLHF.
- InstructGPT (chapter 3) · Introduces RLHF as a three-step process.
- The role of RLVR (chapter 7) · The discussion of RLVR builds on its separate role in verifiable domains.
- Character training (chapter 17) · Character training develops the behavioral qualities introduced here.
- Training a Bradley–Terry reward model (chapter 5) · Reward-model training develops the use of comparisons between responses.
Concepts explained in this chapter
- post-trainingPost-training is the set of optimization and adaptation procedures applied to a pretrained model to improve its usefulness for downstream tasks or alignment objectives.
- reinforcement learning with human feedbackReinforcement learning from human feedback (RLHF) is a model-alignment technique that optimizes behavior using human preference signals, often through a learned reward model or direct preference objective.
- fine-tuningFine-tuning is further training of a pretrained model on a task-specific or behavior-specific objective.
- reward modelA reward model is a learned model that predicts the reward or preference value assigned to an output or behavior.
- foundational modelA foundation model is a broadly pretrained model that can be adapted to support multiple downstream tasks.
- styleResponse style is the characteristic manner in which a model presents content, including presentation and rhetorical qualities such as bluntness or warmth.
- preference tuningPreference optimization is model training that adjusts behavior toward preferred outputs using preference-based objectives or rewards.
- preference dataA pairwise comparison is an evaluation that determines which of two items is preferred.
- contrastive loss functionA contrastive loss is an optimization objective that compares examples to encourage similar or preferred examples to be distinguished from dissimilar or non-preferred ones.
- elicitation theory of post-trainingThe elicitation theory of post-training is the view that post-training primarily extracts and amplifies valuable capabilities already present in a pretrained base model.
- superficial alignment hypothesisThe superficial alignment hypothesis is the claim that pretraining supplies most knowledge and capability while alignment mainly teaches a model a narrow set of interaction formats or styles.
- preferenceA human preference is a person's comparative judgment about the desirability or quality of model behavior or outputs.
- reinforcement learningReinforcement learning is a machine learning paradigm in which an agent learns behavior by optimizing rewards obtained through interaction with an environment.
- three-stage rlhf recipeA three-stage RLHF recipe is a workflow consisting of instruction tuning, reward-model training, and reinforcement-learning optimization.
- Direct Preference OptimizationDirect preference optimization (DPO) directly trains a policy from pairwise preferred and dispreferred outputs relative to a reference policy.
- instruct modelAn instruction-tuned model is a language model adapted through training on instructions paired with desired responses.
- reinforcement learning with verifiable rewardsReinforcement learning with verifiable rewards (RLVR) is reinforcement learning in which rewards are derived from objectively checkable outcomes.
- output formatResponse format is the structural arrangement and presentation convention of a model response.
- over-optimizationReward over-optimization is a condition in which a reward model's score continues to improve while actual quality measured by held-out evaluations or human judgment degrades.
- length biasLength bias is a systematic preference for outputs of particular lengths that can distort model optimization or evaluation.
- proxy objectiveA proxy objective is an optimization objective that approximates a desired goal instead of measuring that goal directly.
- regularizationRegularization is a technique that constrains optimization to reduce undesirable or excessive fitting to an objective.
- verifiable rewardA verifiable reward is a reward signal whose correctness can be checked objectively by a procedure or criterion.
- next-token predictionNext-token prediction is a language-modeling objective that trains a model to predict the token following a given context.
- pretrainingPretraining is the initial large-scale optimization of a model on broad data, commonly through next-token prediction.
- reasoning model trainingReasoning training is the optimization of a model to produce or support more effective multi-step problem-solving behavior.
- alignmentAlignment is the process of shaping a model's behavior and outputs to conform to desired human or task-related objectives.
- reward signalA reward is a scalar evaluative signal indicating the desirability of an action or outcome for an agent.
- benchmarkA benchmark is a standardized set of questions, tasks, or environments designed to measure a specific property of a model.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.