7 Reasoning and inference-time scaling
This chapter covers
- How and why language models are trained to reason
- How reinforcement learning with verifiable rewards (RLVR) builds on RLHF
- Common implementation decisions for RLVR
- Key reasoning models that established best practices
Reasoning models and inference-time scaling enabled a massive step in language model performance at the end of 2024, through 2025, and into the future. Inference-time scaling is the ability to improve model performance by using more computation during generation, such as producing longer reasoning chains or sampling multiple responses. Language models trained to think extensively before answering exploit this property remarkably well. These models, trained with a large amount of reinforcement learning with verifiable rewards (RLVR) [1], still utilize large amounts of RLHF. In this chapter, we review the path that led the AI community to a transformed appreciation for RL’s potential in language models, review the fundamentals of RLVR, highlight key works, and point to the future debates that will define the area in the next few years. A system diagram of RLVR is shown in figure 7.1, to highlight small differences from the other RL methods in this book.
Figure 7.1 RLVR in the form of an RL feedback loop. Instead of a reward model, a verification function is used.
