A reasoning model is a language model optimized to solve a problem through a multi-step reasoning process before producing its answer. Its defining practical property is inference-time scaling: allocating more generation compute—by producing longer reasoning chains or sampling multiple responses—can improve performance. This behavior became especially important for mathematics, coding, science, and other tasks with checkable answers. section 7.0 section 7.5
The usual training recipe combines instruction tuning and RLHF with reinforcement learning from verifiable rewards (RLVR). Rather than relying solely on a learned reward model to judge subjective qualities, RLVR can use a verification function: answer matching, unit tests, code execution, or theorem verification. A simple loop samples several completions, identifies correct outcomes, and updates the policy toward them. section 7.1 section 7.5 section 7.3.1
Reasoning training changes the post-training pipeline in several operational ways:
- Use binary or otherwise definitive correctness signals where a verifier can assess an answer. section 7.1 section 7.5
- Train on appropriately difficult problems; a common filter retains tasks the current model solves roughly 20–80% of the time. section 7.5
- Relax or remove KL penalties when greater exploration is needed, while recognizing that this changes the optimization trade-off. section 7.5
- Reward consistent answer formatting and language, and schedule reasoning length progressively to limit overthinking. section 7.5
From subjective preference to verification
A response explaining opportunity cost requires an evaluator to judge clarity, accuracy, completeness, and helpfulness. In contrast, a mathematics answer or program can often be checked directly against an expected answer or unit tests. This difference makes the reward signal more definitive and lets training focus on completions whose correctness can be verified. section 7.1 section 7.5
Reasoning models are not defined solely by displaying a chain of thought. Earlier methods such as STaR, Quiet-STaR, TRICE, VinePPO, and verifier-feedback approaches explored ground-truth rewards, generated traces, and binary correctness signals. Later large-scale systems established a recipe that retained broad capability while substantially improving performance in verifiable domains; earlier approaches could be smaller in scale or trade general performance for stronger math or coding ability. section 7.3.1 section 7.5
Figure 7.1 RLVR in the form of an RL feedback loop. Instead of a reward model, a verification function is used.
The RLVR feedback loop makes the central architectural change visible: a verification function supplies reward in place of a reward model. section 7.0
Answer the following multiple-choice question by giving the correct answer
letter in parentheses.
Provide CONCISE reasoning for the answer, and make sure to finish the
response with "Therefore, the answer is (ANSWER_LETTER)" where
(ANSWER_LETTER) is one of (A), (B), (C), (D), (E), etc.
Question: {question}
(A) {choice_A}
(B) {choice_B}
(C) ...
Answer the above question and REMEMBER to finish your response with the
exact phrase "Therefore, the answer is (ANSWER_LETTER)" where
(ANSWER_LETTER) is one of (A), (B), (C), (D), (E), etc.
A generation-based evaluation prompt illustrates the expected interface: produce concise reasoning, then emit an exact answer format that can be checked automatically. section 16.1.4
Evaluation must account for this behavior. Reasoning-era prompts commonly request generative chain-of-thought responses and use exact-match checking rather than single-token answer sampling. Temperature is often greater than zero, and explicit formatting can separate thinking tokens from answer tokens so that the final answer remains machine-checkable. section 16.1.4
