Reinforcement learning with verifiable rewards

Reinforcement learning with verifiable rewards (RLVR) is a post-training technique for improving language-model performance where correctness can be checked objectively. Rather than asking a learned reward model to judge subjective qualities such as helpfulness or clarity, RLVR uses a verification function that returns a definitive score, commonly a positive reward for a correct answer and 0 otherwise. It is therefore suited to domains such as mathematics, coding, science, and theorem proving, where answer matching, unit tests, code execution, or formal verification can supply feedback. section 7.1 section 7.5 section 7.3.1

Figure 7.1 RLVR in the form of an RL feedback loop. Instead of a reward model, a verification function is used.

The feedback loop resembles other RL post-training pipelines, but replaces the learned reward model with a verification function that scores generated answers. section 7.0

The practical distinction from RLHF is the source of reward. RLHF must model human preferences over responses whose quality has no single definitive answer. RLVR can directly test an output against a criterion. For a question asking for the sum of primes below 20, a model may place its result in boxed notation; a simple extraction rule can compare that result with 77 and assign reward 1 when they match. The verifier evaluates the outcome, not necessarily the path the model used to reach it. section 7.1 section 7.1

A core RLVR loop is operationally simple:

• Provide prompts with checkable target outcomes, such as correct numerical answers or executable code behavior. section 7.2.3 section 7.5
• Sample multiple candidate answers from the current policy and run the verifier on each output. section 7.5
• Use policy-gradient updates to reinforce correct completions, then repeat over the training problems. section 7.5
• Train repeatedly on a relatively small prompt set when needed to turn behaviors that occur sporadically in a base model into robust ones. section 7.2.3

RLVR is usually a finishing component in a broader post-training recipe, not a replacement for all other stages. Instruction fine-tuning establishes instruction following, while preference tuning and RLHF shape human-preferred behavior. RLVR then boosts performance on verifiable skills. Modern reasoning-model recipes can sequence these stages differently: one documented path uses an initial reasoning cold start, large-scale RLVR, rejection sampling plus supervised fine-tuning, and a final mixed stage combining verifiable rewards with preference-tuning rewards. section 1.1 section 3.2.3 section 3.2.2

Reasoning training connects RLVR to inference-time scaling: spending more generation-time computation through longer reasoning chains or multiple sampled responses can improve results. RL-trained reasoning models learn longer chains associated with stronger performance, rather than merely inheriting the superficial length bias associated with early RLHF. Viable reasoning RL also appears to depend on sufficiently capable pretrained models; broad uptake became associated with leading models from roughly 2024 onward. section 7.0 section 7.5 section 7.2.1

Common implementation choices balance exploration, signal quality, and unwanted behavior:

• Filter for problem difficulty, often retaining problems the model solves about 20–80% of the time. section 7.5
• Remove or relax KL penalties to permit greater exploration away from a reference policy. section 7.5
• Reward consistent formatting or language when answer extraction and reliable verification depend on it. section 7.5
• Use progressive length scheduling to reduce overthinking. section 7.5