3 Training overview
This chapter covers
- Reinforcement learning basics
- How RLHF relates to traditional RL
- An outline of the RLHF tools you’ll learn in this book
- RLHF training recipes of popular models like InstructGPT, Tülu 3, and DeepSeek R1
In this chapter, we provide a cursory overview of reinforcement learning from human feedback (RLHF) training before getting into the specifics later in the book. RLHF, while optimizing a simple loss function, involves training multiple different AI models in sequence and then linking them together in a complex online optimization.
Here, we introduce the core objective of RLHF: optimizing a proxy reward for human preferences with a distance-based regularizer (we also show how it relates to classical RL problems). Then we present canonical recipes that use RLHF to create leading models to show how RLHF fits with other post-training methods. These example recipes will serve as references for later in the book, where we describe different optimization choices you have when doing RLHF, and we will point back to how different key models used different steps in training.
Note To see concrete examples of how model outputs differ at each training stage, see the completions library at https://rlhfbook.com/library.