Chapter 7: Reasoning and inference-time scaling
The introduction defines inference-time scaling and places reasoning models and RLVR in the context of RLHF.
After this chapter you will be able to
- Inference-Time Scaling. Explain that inference-time scaling means improving model performance by using more computation during generation, for example through longer reasoning chains or multiple sampled responses.
- RLVR and RLHF Relationship. Describe how the introduction positions RLVR in reasoning-model training: these models are trained with a large amount of RLVR and still use large amounts of RLHF.
- RLVR and RLHF rewards. Distinguish RLVR's correctness-based scoring from RLHF's learned evaluation of subjective response qualities.
- Forms of answer verification. Describe how a math answer or generated code can be verified, and explain why verifiability does not require one predetermined solution.
- Repeated correctness based training. Trace the section's basic RLVR training cycle and explain the claimed source of improvement.
- RL stability and adoption. Explain the two ways RL training can be unstable and why its growing use in language-model training indicates progress on this longstanding barrier.
- Base model capability threshold. Explain the section's claim that reasoning-focused RL training needed sufficiently capable underlying models.
- Inference-time scaling concept. Explain that inference-time scaling uses more computation during inference to improve downstream task performance, and that RL training is one route to it rather than the only route.
- Productive response length increase. Describe the link between RL-trained models generating more tokens and better downstream performance, and distinguish it from the earlier RLHF length bias that brought only marginal preference gains.
- Answer-focused RLVR objective. Distinguish RLVR’s focus on producing correct answers from standard instruction fine-tuning’s focus on matching supplied completions.
- Repeated RLVR reinforcement. Explain how this section contrasts the amount of training in standard instruction tuning and RLVR, and what repeated RLVR training is intended to accomplish.
- Reasoning model training shift. Explain how reasoning-model recipes retain instruction fine-tuning, RLHF, and RLVR but place much greater emphasis on RLVR. Describe the two core training paths identified here: a large-scale RL run or large-scale instruction tuning on outputs from a model substantially trained with RLVR, called distillation.
- Early trace-based reasoning methods. Describe how STaR, Quiet-STaR, and TRICE used verifiable feedback or generated reasoning traces to improve reasoning, and distinguish their approaches from binary-reward training.
- Verifier rewards and broader performance. Explain how VinePPO and Tülu 3 moved toward rewarding correct answers with PPO-based training, and why Tülu 3’s broader performance mattered in the context of earlier work.
- Difficulty filtering and scheduling. Explain why reasoning-model training often filters problems by difficulty and may change when problems are presented during training.
- Parallel answer selection. Describe the two answer-selection approaches given for parallel test-time compute scaling.
- RL’s central but evolving role. Explain why RL is no longer treated as a minor finishing step in frontier model training, while recognizing that today’s RLVR techniques are not final answers.
What it builds on
- The role of reinforcement learning in RLHF (chapter 6) · Introduces the role of reinforcement learning in RLHF.
- Training a Bradley–Terry reward model (chapter 5) · The role of a learned reward model in scoring preferences.
- Canonical training recipes (chapter 3) · The broad sequence of language-model post-training stages.
- DeepSeek R1 (chapter 3) · The earlier overview of the DeepSeek R1 training recipe.
- Proximal policy optimization (chapter 6) · Introduces proximal policy optimization, the algorithm named in these training approaches.
- Group relative policy optimization (chapter 6) · Group relative policy optimization and its policy-gradient context.
- Asynchronous RL systems (chapter 6) · Asynchronous RL systems.
Section by section
Introduction
The introduction defines inference-time scaling and places reasoning models and RLVR in the context of RLHF.
- Inference-Time Scaling: Explain that inference-time scaling means improving model performance by using more computation during generation, for example through longer reasoning chains or multiple sampled responses.
Why it matters: It explains why spending more computation on an answer can improve a reasoning model’s performance. - RLVR and RLHF Relationship: Describe how the introduction positions RLVR in reasoning-model training: these models are trained with a large amount of RLVR and still use large amounts of RLHF.
Why it matters: RLVR is presented as part of the training picture, not as a complete replacement for RLHF.
Key terms: inference-time scaling, reasoning chains, reinforcement learning with verifiable rewards (RLVR), RLHF
7.1 The role of RLVR
This section explains how reinforcement learning with verifiable rewards (RLVR) scores answers, how its training cycle improves reasoning models, and how those models can use thinking tokens before answering.
- RLVR and RLHF rewards: Distinguish RLVR's correctness-based scoring from RLHF's learned evaluation of subjective response qualities.
Why it matters: The kind of feedback available determines whether training needs a learned preference judge or can check answers directly. - Forms of answer verification: Describe how a math answer or generated code can be verified, and explain why verifiability does not require one predetermined solution.
Why it matters: RLVR applies to more than questions with one known answer: a test can also check whether a proposed solution works. - Repeated correctness based training: Trace the section's basic RLVR training cycle and explain the claimed source of improvement.
Why it matters: This simple feedback cycle is the section's central intuition for how verifiable rewards can improve a model. - Reasoning model thinking tokens: Describe the visible difference between the ordinary model and reasoning model in the poem example, without treating the example as a detailed account of how reasoning works.
Why it matters: The example makes the extra generation associated with a reasoning model observable. - RL within training stages: Explain the section's cake analogy for the relative roles of self-supervised learning, supervised fine-tuning, and scaled-up RL.
Why it matters: The analogy puts reasoning-focused RL in proportion to the earlier stages that provide the model's broad capabilities.
Key terms: RLVR, RLHF, reward model (RM), verification functions, answer extraction, unit tests, ground-truth answer, partial credit, gradient steps, verifiable rewards
7.2 The origins of new reasoning models
7.2.1 Why does RL work now?
The section explains why RL training for language-model reasoning has become viable: improved stability, accessible tooling, and sufficiently capable base models.
- RL stability and adoption: Explain the two ways RL training can be unstable and why its growing use in language-model training indicates progress on this longstanding barrier.
Why it matters: It explains why a method long considered unreliable is now being used for reasoning models. - Accessible RL training tools: Describe how available tools have lowered the barrier to training language models with RLVR and supported more research.
Why it matters: Available tooling helps explain why more groups can investigate RL-based reasoning training. - Base model capability threshold: Explain the section's claim that reasoning-focused RL training needed sufficiently capable underlying models.
Why it matters: It identifies base-model capability, not just the RL method, as part of why reasoning training works now.
Key terms: stability, RL training, verifiable rewards, pretrained base model, RLVR, TRL, Open Instruct, veRL, OpenRLHF, underlying capability
7.2.2 RL training vs. inference-time scaling
This section explains how inference-time scaling relates to RL-trained reasoning models and why longer responses matter only when they improve performance.
- Inference-time scaling concept: Explain that inference-time scaling uses more computation during inference to improve downstream task performance, and that RL training is one route to it rather than the only route.
Why it matters: It keeps the broader idea of spending compute at inference distinct from one way of training models to do so. - Productive response length increase: Describe the link between RL-trained models generating more tokens and better downstream performance, and distinguish it from the earlier RLHF length bias that brought only marginal preference gains.
Why it matters: More generated tokens are worth the extra inference cost only when they accompany better performance.
Key terms: inference-time scaling, test-time scaling, RL training, downstream performance, length bias, inference-time compute
7.2.3 The future (beyond reasoning) of RLVR
This section contrasts answer-focused RLVR with standard instruction fine-tuning and explains how repeated RLVR training can make initially sporadic successful behaviors more reliable.
- Answer-focused RLVR objective: Distinguish RLVR’s focus on producing correct answers from standard instruction fine-tuning’s focus on matching supplied completions.
Why it matters: The distinction explains why RLVR can be more directly aligned with a developer’s performance goal. - Repeated RLVR reinforcement: Explain how this section contrasts the amount of training in standard instruction tuning and RLVR, and what repeated RLVR training is intended to accomplish.
Why it matters: The repetition explains how RLVR can strengthen a capability the base model exhibits only occasionally.
Key terms: RLVR, supervised fine-tuning, prompts and completions, queries and correct answers, epochs, base model, robust behaviors
7.3 Understanding reasoning training methods
This section explains how reasoning-model training changes familiar post-training recipes by emphasizing RLVR and varying the order of training steps.
- Reasoning model training shift: Explain how reasoning-model recipes retain instruction fine-tuning, RLHF, and RLVR but place much greater emphasis on RLVR. Describe the two core training paths identified here: a large-scale RL run or large-scale instruction tuning on outputs from a model substantially trained with RLVR, called distillation.
Why it matters: This distinction helps you recognize what changes—and what stays familiar—in reasoning-model training recipes.
Key terms: instruction fine-tuning, RLHF, RLVR, distillation
7.3.1 Reasoning research before OpenAI o1 or DeepSeek R1
This section traces pre-o1 and pre-DeepSeek R1 approaches to training reasoning in verifiable domains and contrasts their feedback methods and results.
- Early trace-based reasoning methods: Describe how STaR, Quiet-STaR, and TRICE used verifiable feedback or generated reasoning traces to improve reasoning, and distinguish their approaches from binary-reward training.
Why it matters: These approaches show that verifiable feedback was being used to train reasoning before the newer reasoning models. - Verifier rewards and broader performance: Explain how VinePPO and Tülu 3 moved toward rewarding correct answers with PPO-based training, and why Tülu 3’s broader performance mattered in the context of earlier work.
Why it matters: Correct-answer rewards were already in use; the section highlights both the limits of earlier work and the importance of retaining general performance.
Key terms: STaR, Quiet-STaR, TRICE, ground-truth reward signals, chain-of-thought reasoning, traces, VinePPO, Tülu 3, PPO, binary rewards
7.3.2 Early reasoning models
Introduces a table of foundational reasoning-model reports following DeepSeek R1 and defines abbreviations used with it.
7.3.3 Common practices in training reasoning models
This section surveys practical choices for training reasoning models, scaling their inference, and retaining useful capabilities, while cautioning that the examples do not establish an optimal recipe.
- Difficulty filtering and scheduling: Explain why reasoning-model training often filters problems by difficulty and may change when problems are presented during training.
Why it matters: The choice and timing of training problems affect whether the model receives a useful learning signal. - Policy optimization adjustments: Describe the section's examples of changing KL constraints, policy-gradient clipping, and loss normalization during reasoning training.
Why it matters: These are distinct ways recipes adjust how policy updates are constrained or scaled. - Long rollout throughput: Explain why longer, variably sized reasoning completions motivate asynchronous updates or different batch arrangements.
Why it matters: A training recipe must account for the compute cost of waiting on unusually long reasoning responses. - Behavior and length controls: Describe how auxiliary rewards and length controls shape reasoning-model behavior during training.
Why it matters: Correct answers are not the only training concern: the form, language, and length of reasoning also affect the resulting model. - Parallel answer selection: Describe the two answer-selection approaches given for parallel test-time compute scaling.
Why it matters: Inference-time improvement can come from generating and selecting among several attempts, not just from further training. - Capabilities beyond text reasoning: State the section's findings about multimodal performance and user-controlled reasoning length.
Why it matters: Reasoning training need not come at the expense of other capabilities, and some models offer users control over reasoning effort.
Key terms: RLVR, offline difficulty filtering, per-batch online filtering, difficulty curricula, KL penalty, relaxed policy-gradient clipping, GRPO, loss normalization, batch level, group level
7.4 Looking ahead
The section places RL at the center of frontier reasoning-model training while emphasizing that current RLVR practices are provisional and increasingly subject to systematic study.
- RL’s central but evolving role: Explain why RL is no longer treated as a minor finishing step in frontier model training, while recognizing that today’s RLVR techniques are not final answers.
Why it matters: This distinction helps you treat RL as central without treating today’s training recipe as permanent. - Systematic study of reasoning training: Describe how researchers are beginning to study reasoning training more systematically through open training records and investigations of scaling relationships.
Why it matters: These efforts show how researchers can test and refine practices that are still evolving.
Key terms: RL, RLVR, difficulty filtering, format rewards, OLMo 3 Think, checkpoints, scaling properties, compute, data, performance
7.5 Summary
Where this is used later
- Contamination (chapter 16) · the role of RL in training for verifiable rewards
- Why many external evaluation comparisons are unreliable (chapter 16) · The distinction between model training and inference-time scaling.
- Implementation details (chapter 13) · Reasoning-model training methods
Concepts explained in this chapter
- reinforcement learning with verifiable rewardsReinforcement learning with verifiable rewards (RLVR) is reinforcement learning in which rewards are derived from objectively checkable outcomes.
- reasoning model trainingReasoning training is the optimization of a model to produce or support more effective multi-step problem-solving behavior.
- reasoning modelA reasoning language model is a language model optimized to solve problems through explicit or implicit multi-step reasoning before producing a final answer.
- verificationA verification function evaluates a candidate answer or behavior against an objectively checkable correctness criterion.
- inference-time compute scalingInference-time scaling improves model performance by allocating additional computation during inference, such as generating and evaluating more candidate reasoning paths.
- difficulty filteringDifficulty filtering is a training-data selection technique that retains or emphasizes examples according to their estimated difficulty.
- DAPODynamic sampling policy optimization (DAPO) is a GRPO variant that combines dynamic sampling, decoupled clipping, token-level optimization, and length-aware penalties for reasoning-model training.
- asynchronous trainingAsynchronous reinforcement learning is an RL training arrangement in which rollout generation and policy optimization proceed concurrently on separate workers with periodically synchronized parameters.
- loss normalizationLoss normalization is the rescaling of optimization losses or advantage estimates using statistics computed over a specified set of examples.
- online filteringOnline difficulty filtering is the selection of training problems during optimization according to their current difficulty or learning signal.
- Proximal Policy OptimizationPPO is a policy-gradient reinforcement-learning algorithm that constrains policy updates to remain near a reference policy.
- reinforcement learningReinforcement learning is a machine learning paradigm in which an agent learns behavior by optimizing rewards obtained through interaction with an environment.
- verifiable rewardA verifiable reward is a reward signal whose correctness can be checked objectively by a procedure or criterion.
- verifiable taskA verifiable domain is a problem area in which candidate outputs can be objectively checked for correctness.
- gradient descentA gradient update, or gradient step, changes model parameters using the gradient of an optimization objective.
- reward modelA reward model is a learned model that predicts the reward or preference value assigned to an output or behavior.
- thinking tokenA thinking token is a generated language-model token belonging to an intermediate reasoning sequence produced before a final response.
- response-formatting rewardA format reward is a reward signal assigned according to whether a model output follows a required structural presentation convention.
- chain-of-thought promptingA reasoning trace is a sequence of generated textual intermediate steps used while solving a problem before reaching an answer.
- code executionCode execution is the process of running source code to obtain its computational behavior or result.
- progressive length schedulingProgressive length scheduling is a training procedure that changes the targeted or permitted output length over successive optimization stages.
- distillationDistillation is a model-training technique in which a student model learns to reproduce the outputs or behavior of a teacher model.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.