Chapter 3: Training overview
This introduction frames RLHF training as a multi-model process centered on optimizing a proxy for human preferences while limiting distance from a reference.
After this chapter you will be able to
- RLHF training objective. State that RLHF optimizes a proxy reward for human preferences together with a distance-based regularizer.
- Agent policy and environment. Describe how an RL agent chooses actions from a policy given the current state, receives reward, and encounters states governed by environment dynamics.
- Trajectory probability components. Explain what a trajectory contains and how its probability arises from the initial state, policy action choices, and environment transitions.
- Expected discounted return objective. Describe the finite-episode RL objective as maximizing expected return under a policy, and explain how the horizon and discount factor treat future rewards and continuing tasks.
- Thermostat RL components. Describe the thermostat’s state, actions, reward, policy, and transitions, including which parts of the room’s behavior the agent cannot control.
- Trial-and-error policy improvement. Trace how the thermostat moves from random heater choices toward a more rewarding policy through repeated interaction.
- CartPole state and feedback. Describe CartPole’s four state variables, left-or-right force actions, per-step reward, and episode-ending conditions.
- CartPole transitions and learning. Explain how actions advance CartPole’s state and why repeated episodes lead from random pushes to corrective pushes that increase cumulative reward.
- Learned preference reward. Distinguish RLHF's learned reward model from a standard RL environmental reward function, including the flexibility and implementation cost of that change.
- Prompt completion episode. Describe why a fixed prompt and its completion form a complete RLHF episode rather than a chain of state transitions.
- Sequence reward without discounting. Explain why this setup attributes reward to the whole generated response and usually uses a discount factor of one, and describe the resulting simplification of the optimization.
- Pretrained Model Fine-Tuning. Explain why RLHF refines a pretrained model rather than learning from a random policy, and why that starting point makes regularization important.
- KL Penalty Tradeoff. Describe how a KL divergence penalty constrains the RLHF policy relative to a reference policy, and explain how beta changes the balance between staying close and pursuing reward.
- Post-training tool roles. Distinguish reward modeling, rejection sampling, policy gradients, and direct alignment algorithms by what each uses and how it improves or selects model outputs.
- Instruction tuning and training order. Explain what instruction fine-tuning teaches through imitation and where it sits in the training sequence described here.
- RL value despite complexity. Explain why RL-based post-training remains attractive even though instruction tuning, rejection sampling, and DPO-like methods are simpler to implement.
- Changing post-training recipes. Describe the general trend toward more optimization steps, more training algorithms, and more diverse datasets and evaluations in post-training.
- InstructGPT three-stage training recipe. Describe how instruction tuning, reward-model training, and RLHF build on a base language model, and explain what each stage contributes.
- Tülu 3 training stages. Describe Tülu 3's three stages in order and what each is meant to improve: instruction following and foundational skills, conversational quality and those skills, then core skills such as mathematics while maintaining overall performance.
- DeepSeek R1 training sequence. Trace the DeepSeek R1 recipe from its cold start through large-scale RL, rejection sampling and SFT, and final mixed RL training.
What it builds on
- RLHF in three steps (chapter 1) · Introduces RLHF as a three-step process.
Section by section
Introduction
This introduction frames RLHF training as a multi-model process centered on optimizing a proxy for human preferences while limiting distance from a reference.
- RLHF training objective: State that RLHF optimizes a proxy reward for human preferences together with a distance-based regularizer.
Why it matters: This is the objective that the chapter’s later formulation will unpack. - Multi-model training process: Describe the contrast between RLHF’s simple loss function and its broader training process: multiple AI models are trained in sequence and linked in a complex online optimization.
Why it matters: It explains why understanding RLHF takes more than knowing its objective.
Key terms: proxy reward, human preferences, distance-based regularizer, loss function, online optimization
3.1 Problem formulation
This section establishes the standard reinforcement-learning setup and objective that RLHF optimization builds on.
- Agent policy and environment: Describe how an RL agent chooses actions from a policy given the current state, receives reward, and encounters states governed by environment dynamics.
Why it matters: These parts define the RL setup on which the chapter's account of RLHF builds. - Trajectory probability components: Explain what a trajectory contains and how its probability arises from the initial state, policy action choices, and environment transitions.
Why it matters: The trajectory distribution describes which complete experiences are likely under a policy. - Expected discounted return objective: Describe the finite-episode RL objective as maximizing expected return under a policy, and explain how the horizon and discount factor treat future rewards and continuing tasks.
Why it matters: This objective states what it means to improve a policy before considering particular optimization methods.
Key terms: agent, policy, state, action, reward, transition (dynamics), trajectory, trajectory distribution, initial state distribution, transition probability
3.1.1 A simple example: The thermostat
The thermostat example makes the parts of a reinforcement-learning task and its trial-and-error learning loop concrete.
- Thermostat RL components: Describe the thermostat’s state, actions, reward, policy, and transitions, including which parts of the room’s behavior the agent cannot control.
Why it matters: The example gives you concrete meanings for the parts of an RL task. - Trial-and-error policy improvement: Trace how the thermostat moves from random heater choices toward a more rewarding policy through repeated interaction.
Why it matters: This shows how reward can guide a policy without supplying the agent with the right rule at the start.
Key terms: state, action, reward, policy, transition, trial and error, episodes, RL loop
3.1.2 Classic RL example: CartPole
CartPole illustrates the standard RL setup through a continuous-state control task in which an agent learns to balance a pole with corrective pushes.
- CartPole state and feedback: Describe CartPole’s four state variables, left-or-right force actions, per-step reward, and episode-ending conditions.
Why it matters: These elements make CartPole a concrete example of states, actions, rewards, and episode termination. - CartPole transitions and learning: Explain how actions advance CartPole’s state and why repeated episodes lead from random pushes to corrective pushes that increase cumulative reward.
Why it matters: It connects CartPole’s physical motion to the RL objective of improving reward through interaction.
Key terms: state, action, reward, episode, policy, dynamics/transition, deterministic, cumulative reward
3.1.3 Manipulating the standard RL setup
This section explains how RLHF adapts the standard reinforcement learning setup for language models by using a learned reward model and treating a prompt and its completion as a single, undiscounted episode.
- Learned preference reward: Distinguish RLHF's learned reward model from a standard RL environmental reward function, including the flexibility and implementation cost of that change.
Why it matters: This distinction explains where the reward signal in RLHF comes from. - Prompt completion episode: Describe why a fixed prompt and its completion form a complete RLHF episode rather than a chain of state transitions.
Why it matters: This single-turn structure is a major reason the RLHF setup differs from a traditional sequential RL environment. - Sequence reward without discounting: Explain why this setup attributes reward to the whole generated response and usually uses a discount factor of one, and describe the resulting simplification of the optimization.
Why it matters: It connects the prompt–completion episode to the simpler objective used for RLHF.
Key terms: reward model, human preferences, environmental reward function, prompt, completion, episode, rollout, state transitions, response-level reward, discount factor
3.1.4 Fine-tuning and regularization
This section explains why RLHF fine-tunes an already-capable model with regularization and how a KL penalty limits movement from its starting policy.
- Pretrained Model Fine-Tuning: Explain why RLHF refines a pretrained model rather than learning from a random policy, and why that starting point makes regularization important.
Why it matters: The model’s existing capabilities explain why RLHF must balance improvement against drift. - KL Penalty Tradeoff: Describe how a KL divergence penalty constrains the RLHF policy relative to a reference policy, and explain how beta changes the balance between staying close and pursuing reward.
Why it matters: This tradeoff is how the section frames improving reward while guarding against over-optimization.
Key terms: pretrained base model, initial policy, regularization, Kullback–Leibler (KL) divergence, reference policy, KL budget, beta
3.1.5 Optimization tools
This section identifies the main post-training optimization tools and explains how instruction fine-tuning fits into their usual training sequence.
- Post-training tool roles: Distinguish reward modeling, rejection sampling, policy gradients, and direct alignment algorithms by what each uses and how it improves or selects model outputs.
Why it matters: These distinctions show which approaches use a learned reward signal and which use preferences directly. - Instruction tuning and training order: Explain what instruction fine-tuning teaches through imitation and where it sits in the training sequence described here.
Why it matters: It explains why the other optimization options are presented as additions to, rather than replacements for, instruction fine-tuning.
Key terms: reward modeling, rejection sampling, policy gradients, direct alignment algorithms, pairwise preference data, scalar reward, instruction fine-tuning, preselected examples, question-answer format
3.1.6 Subtle advantages of RL in post-training language models
This section explains why RL remains valuable for post-training language models despite simpler, less infrastructure-intensive alternatives.
- RL value despite complexity: Explain why RL-based post-training remains attractive even though instruction tuning, rejection sampling, and DPO-like methods are simpler to implement.
Why it matters: This explains why practitioners continue to invest in RL rather than choosing only simpler post-training methods. - Targeted improvements without broad loss: Describe the kinds of improvements RL stages can make and how targeted RL training can preserve broader capabilities.
Why it matters: Targeted gains without losing broad capabilities are a practical reason to use RL in post-training.
Key terms: RL-based methods, instruction tuning, direct preference optimization (DPO), RL losses, rough edges, robust, surgically, prompt distribution, Tülu 3
3.2 Canonical training recipes
This section explains how canonical RLHF and post-training recipes reflect their time and how post-training recipes have grown more complex.
- Changing post-training recipes: Describe the general trend toward more optimization steps, more training algorithms, and more diverse datasets and evaluations in post-training.
Why it matters: This trend provides context for comparing the recipes that follow. - Recipes in historical context: Explain why canonical recipes reflect their time and what happens to the difficulty of reproducing their characteristics as they age.
Why it matters: A canonical recipe is a snapshot of its era, not a permanent measure of how much training a model needs.
Key terms: post-training, optimization steps, training algorithms, training datasets, evaluations, canonical recipes, data practices, model abilities
3.2.1 InstructGPT
Describes the three-stage InstructGPT post-training recipe and the role of each stage.
- InstructGPT three-stage training recipe: Describe how instruction tuning, reward-model training, and RLHF build on a base language model, and explain what each stage contributes.
Why it matters: This recipe provides a baseline for understanding the training stages and how later recipes differ.
Key terms: base language model, instruction tuning, reward model, RLHF, instruction-tuned checkpoint
3.2.2 Tülu 3
The section places Tülu 3 in the context of increasingly multistage post-training and explains what each of its three stages contributes.
- Modern post-training complexity: Explain that modern post-training can involve numerous model versions and training iterations, rather than a short, fixed sequence of steps. Note that full training details for leading models are often not public.
Why it matters: This context explains why Tülu 3 is presented as a useful open example rather than a complete picture of every leading model's training. - Tülu 3 training stages: Describe Tülu 3's three stages in order and what each is meant to improve: instruction following and foundational skills, conversational quality and those skills, then core skills such as mathematics while maintaining overall performance.
Why it matters: The recipe shows how different post-training stages can make distinct contributions to a model's behavior and skills.
Key terms: post-training, training stages, training iterations, Tülu 3, instruction tuning, on-policy preference data, RLVR
3.2.3 DeepSeek R1
This section outlines DeepSeek R1’s staged reasoning-model post-training recipe and notes a common variation that uses polished instruction data before RL.
- DeepSeek R1 training sequence: Trace the DeepSeek R1 recipe from its cold start through large-scale RL, rejection sampling and SFT, and final mixed RL training.
Why it matters: The sequence shows how reasoning-focused training is turned into a general-purpose model and then polished. - Early instruction tuning variation: Describe the variation in which polished instruction data provides a strong SFT starting point before RL.
Why it matters: This variation shows that the ordering and preparation of training data are not fixed across reasoning-model recipes.
Key terms: cold start, R1-Zero, RLVR, rejection sampling, SFT, mixed reinforcement learning, instruction datasets, chain-of-thought sequences, RL
3.3 Summary
Where this is used later
- The role of reinforcement learning in RLHF (chapter 6) · Uses the agent-and-policy framing to explain RL's role in RLHF.
- (chapter intro) (chapter 5) · Assumes the basic RL setup in which an agent receives a reward signal.
- Deriving the policy gradient (chapter 6) · Assumes the RL formulation in terms of policies, states, actions, and rewards.
- Training a Bradley–Terry reward model (chapter 5) · Develops the learned reward model used to score preferences.
- Loss aggregation tradeoffs (chapter 6) · Assumes that the standard RL setup can be adapted to language-model training.
- KL divergence in RL optimization (chapter 15) · Motivates constraining the fine-tuned policy relative to a reference model.
- Collecting preference data (chapter 11) · The idea that post-training proceeds through model fine-tuning stages.
- Policy gradient algorithms (chapter 6) · Develops policy gradient algorithms.
Concepts explained in this chapter
- trajectoryAn episode is a finite sequence of interaction steps experienced by an agent during one trial.
- reinforcement learning with human feedbackReinforcement learning from human feedback (RLHF) is a model-alignment technique that optimizes behavior using human preference signals, often through a learned reward model or direct preference objective.
- fine-tuningFine-tuning is further training of a pretrained model on a task-specific or behavior-specific objective.
- reward modelA reward model is a learned model that predicts the reward or preference value assigned to an output or behavior.
- post-trainingPost-training is the set of optimization and adaptation procedures applied to a pretrained model to improve its usefulness for downstream tasks or alignment objectives.
- reinforcement learningReinforcement learning is a machine learning paradigm in which an agent learns behavior by optimizing rewards obtained through interaction with an environment.
- policyA policy is a function that maps an environment state to a probability distribution over actions.
- reward signalA reward is a scalar evaluative signal indicating the desirability of an action or outcome for an agent.
- discount factorA discount factor is a scalar between zero and one that weights future rewards relative to nearer rewards.
- rejection samplingRejection sampling is a sampling method that accepts or filters candidate outputs according to a specified criterion or scoring procedure.
- regularizationRegularization is a technique that constrains optimization to reduce undesirable or excessive fitting to an objective.
- Kullback–Leibler divergenceForward KL divergence is KL divergence from a target distribution to an approximating distribution, with expectation taken under the target distribution.
- direct alignment algorithmA direct alignment algorithm optimizes a model directly from preference data without learning an intermediate reward model.
- policy gradient methodPolicy gradient is a reinforcement-learning technique that optimizes policy parameters by estimating gradients of expected reward from sampled trajectories.
- trajectory distributionA trajectory distribution is the probability distribution over complete interaction sequences induced by an initial-state distribution, policy, and environment dynamics.
- KL regularizationA KL penalty is a regularization term proportional to the Kullback–Leibler divergence between an optimized distribution or policy and a reference distribution or policy.
- reinforcement learning with verifiable rewardsReinforcement learning with verifiable rewards (RLVR) is reinforcement learning in which rewards are derived from objectively checkable outcomes.
- reference modelA reference model is a frozen copy of a policy that provides a baseline distribution for measuring or penalizing policy deviation.
- on-policy methodOn-policy optimization, or on-policy learning, updates a policy using data generated by that same policy or its current update distribution.
- completionA candidate completion is a generated language-model output considered for acceptance, ranking, or further training.
- multi-armed banditA bandit problem is a sequential decision problem in which an action produces an outcome without a multi-step state-transition sequence.
- promptA prompt is an input text or instruction provided to a language model to elicit a response.
- gradient descentA gradient update, or gradient step, changes model parameters using the gradient of an optimization objective.
- synthetic dataSynthetic data is data generated by an algorithmic or model-based process rather than directly collected from the target phenomenon.
- returnCumulative reward is the aggregate of rewards collected across an episode or other interaction sequence.
- instruction dataAn instruction dataset is a collection of prompts or instructions paired with target responses for supervised model adaptation.
- preferenceA human preference is a person's comparative judgment about the desirability or quality of model behavior or outputs.
- proxy reward functionA proxy reward is a reward signal that approximates a desired human or task objective rather than measuring it directly.
- preference dataA pairwise comparison is an evaluation that determines which of two items is preferred.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.