Part 3. Data and preferences

With the core methods in hand, this part turns to the data that drives RLHF. RLHF is fundamentally a data problem: the quality of a model’s behavior is bounded by the quality of the preferences it learns from. The same holds for post-training gener-ally. These chapters explore why RLHF problems don’t have one perfect solution, what preferences actually are, how data is collected, and how AI-generated synthetic data is increasingly replacing human annotation in modern training pipelines.

Chapter 10 steps back from the technical machinery to motivate the broader context of RLHF and why it is such a crucial problem. It asks questions such as, what are preferences, and why should we expect them to improve models? This connects RLHF to fields well beyond machine learning, including philosophy and economics. Chapter 11 covers the practical side of preference data: how it is collected, curated, and shaped to train models. Chapter 12 examines the growing role of synthetic data at the frontier of AI development, covering topics such as distillation, constitutional AI, rubrics as rewards, and more.

After reading this part of the book, you’ll understand not just how models are trained, but what they’re trained on—and why the data choices matter as much as the algorithms.