Part 4. Applications and advanced topics

RLHF in practice is a messy, complicated process that unfolds at the cutting edge of AI development. This part of the book covers the remaining pieces needed to understand whether post-training actually produces a useful, well-behaved model. These chapters span a wider range of subjects, with the through line being the complexity and insight that come from the time-consuming exploration of rabbit holes to get model training just right.

Chapter 13 introduces tool use and function calling, which teach models to interact with external APIs and execute code—capabilities behind some of the most popular agentic models. Chapters 14 and 15 tackle over-optimization and regularization as two sides of the same coin: why optimizing against a learned reward inevitably exploits its weaknesses, and the techniques—most notably, Kullback–Leibler (KL) penalties—used to keep that optimization in check. Chapter 16 surveys the evolving landscape of evaluation and benchmarks, tackling the difficult question of how to measure whether post-training actually makes the model better. The book closes with chapter 17 on product, user experience, and model character: how RLHF shapes the personality and behavior of deployed models, and how AI laboratories tailor training to match their products.

These chapters bridge the gap between the methods and the messy reality of building AI systems that people use.