Part 2. Core training methods

This is the foundation of this book, detailing the methods used to perform RLHF and post-training more broadly. The chapters in this part focus on the key intuitions and steps for applying these methods, with comparisons between algorithmic variants and core implementation details. Far and away the most technically advanced chapter is chapter 6, which covers the policy-gradient reinforcement learning (RL) algorithms used to optimize language models for RLHF (and the reasoning models trained with reinforcement learning with verifiable rewards (RLVR), discussed in chapter 7). This part of the book is about the action of changing models—it is where much of the magic of modern language models comes to life.