Chapter 7: Reinforcement learning: From policy gradients to GRPO
Introduces reinforcement-learning post-training as a way to turn a text-predicting base model toward rewarded reasoning behavior, using DeepSeek-R1 as the example.
After this chapter you will be able to
- Reward-Based Post-Training. Explain how reinforcement-learning post-training differs from training a base model to predict text: the model tries complete responses, receives rewards, and is updated to make rewarded behavior more likely.
- DeepSeek-R1 Reasoning Result. Describe the chapter’s claim that training a large base model with GRPO and verifiable rewards on objectively checkable tasks led it to produce longer, more self-corrective solutions, rather than simply memorizing human-written reasoning examples.
- Agent interaction and reward. Describe how an agent observes a state, takes an action, receives a reward, and observes a new state, learning over repeated interactions which actions tend to earn higher rewards.
- Final answer rewards for reasoning. Explain why rewarding a correct final answer can be useful for math reasoning when the desired sequence of intermediate tokens is unknown.
- Agent–environment interaction loop. Trace how the cleaning robot observes the room, chooses an action, and receives a reward after the room changes, then repeats the cycle.
- Policy and reward-driven training. Explain what the robot’s policy does and how training changes a neural-network policy in relation to long-term reward.
- Language Generation as RL. Trace how a prompt and generated tokens form states, how each sampled next token acts as an action, and when the completed response receives a reward.
- Sampling During RL Rollouts. Explain why reinforcement-learning rollouts sample from the model’s probability distribution instead of always selecting the most likely token.
- Sparse Rewards and Credit Assignment. Explain that a verifier may reward a completed answer without scoring each token, so the reward does not reveal which token or mistake caused the outcome. Describe how policy-gradient methods respond by updating the probabilities of all sampled actions in proportion to that outcome.
- Reward-driven token probabilities. Explain how a policy-gradient method relates a rewarded rollout to changes in an LLM’s token probabilities, identifying the policy and its parameters.
- Sampling for exploration. Explain why a rollout samples from the next-token probabilities instead of always choosing the highest-probability token.
- Sampled response trajectories. Describe how sampled tokens make up a response trajectory and what probabilities the policy-gradient update changes.
- Reward-weighted token updates. Explain how the log-probability gradient and reward affect the probability of a sampled token, and state that this simplified update uses raw reward without a baseline.
- Prompt-relative response quality. Explain why a raw reward may not show how good a completion is relative to other completions for the same prompt.
- Advantages and baseline stability. Describe how an advantage compares an outcome with a baseline, and why using a baseline can make training updates less noisy.
- PPO update stability. Explain why PPO limits changes to a policy after collecting rollouts, and describe how its probability ratio and clipping help do so.
- PPO training components. Describe the four components of PPO for LLM post-training: a trainable policy LLM, a reward signal, a value model, and a frozen reference model. Explain that the value model estimates expected reward to provide a baseline for advantage calculation.
- Reference model drift control. Explain why PPO keeps a frozen pre-RL reference model and uses a KL penalty when the trained policy moves too far from it.
- Learned Value Function. Explain how the value model learns to predict the expected return from a partial response using states observed during rollouts and their eventual returns.
- Reference policy constraint. Explain why an imperfect reward function calls for a reference-policy constraint and how a token-level KL penalty discourages the trained policy from moving too far away.
- Group-relative reward baseline. Explain how GRPO samples multiple completions for one prompt, rewards each, and normalizes their rewards so that better-than-group completions have positive advantages and worse-than-group completions have negative advantages, without a learned value model.
- Group-relative advantage calculation. Explain how rewards from multiple completions of the same prompt are turned into advantages by comparing each reward with the group mean and accounting for the group's reward variation.
- Full GRPO objective components. Explain that GRPO uses a group-relative advantage while retaining PPO’s clipped policy ratio and a KL term against a frozen reference policy. Distinguish what each term does.
- GRPO cost advantage over PPO. Explain that GRPO's cost advantage comes from eliminating the learned value model and its training path, not from sampling fewer completions.
- Deterministic reward signals. Explain that RLVR assigns rewards using deterministic checkers instead of a learned reward model, and give an example of a check for math, code, or a symbolic task.
- Limits of verifiable rewards. Distinguish tasks with objectively verifiable correctness from open-ended assistant tasks, and explain why the latter still need other alignment signals.
- Correctness and formatting balance. Explain how a verifier can reward correct answers without rejecting harmless differences in how they are written. Describe the toy math check and why a production math verifier may need more robust comparisons.
- Verifier rewards shape behavior. Explain how the checks included in a reward function can change the model's responses, using the correctness and format rewards described for DeepSeek-R1-Zero.
- Limits of verifiable rewards. Distinguish tasks suited to deterministic verification from tasks that require preference judgments, and explain why reward models remain useful in a hybrid reward design.
- R1-Zero training setup. Describe how DeepSeek-R1-Zero starts from DeepSeek-V3-Base and applies large-scale reinforcement learning with rewards for verifiable reasoning tasks, without first training on supervised chain-of-thought examples.
- Emerging reasoning behavior. Explain how the model’s use of more tokens on difficult problems can provide room to try, check, and revise an approach, and what the reported 'aha moment' illustrates. Describe what the <think> tag denotes in that example.
- Limits of R1-Zero findings. Distinguish the finding that reasoning-like behavior can emerge under RL pressure on verifiable tasks from claims that any reward guarantees reasoning or that R1-Zero is a polished model.
- DeepSeek-R1 model distinctions. Distinguish DeepSeek-R1-Zero, DeepSeek-R1, and DeepSeek-R1-Distill by their training approach and purpose.
- R1-Zero to R1 data flow. Distinguish the flow of training data from R1-Zero to R1 from a continuation of R1-Zero's parameters. Explain why the sampled reasoning traces need filtering and refinement.
- Cold start and reasoning RL. Describe how cold-start supervised examples establish a readable reasoning format before reasoning-oriented RL, and how that RL stage addresses language mixing.
- Filtered data and assistant behavior. Explain how rejection sampling and supervised fine-tuning use filtered outputs and broader instruction data to restore general assistant behavior.
- Final hybrid reward signals. Distinguish the reward sources in R1's final RL stage for verifiable tasks and open-ended helpfulness and harmlessness tasks.
- Cold-start and RL roles. Explain why DeepSeek-R1 uses curated supervised reasoning examples before its main RL stage, and how supervised fine-tuning and RL address different needs.
- Rejection Sampling Data Construction. Describe how rejection sampling generates supervised training data by sampling candidate responses, retaining those that meet correctness and quality criteria, and discarding the rest.
- Filtering Criteria and Pipeline Flow. Explain why the pipeline filters an RL-trained model's outputs and how those outputs contribute to R1 training rather than simply transferring weights from the pure-RL experiment.
- Limits of narrow rewards. Explain why optimizing heavily for math and code verifiers can improve verifiable reasoning without producing a polished general assistant.
- Broad supervised data mix. Describe how DeepSeek-R1 uses mixed supervised data after rejection sampling to help preserve general assistant behavior, and distinguish that step from relying on GRPO and verifiable rewards alone.
- Task-matched hybrid rewards. Explain why DeepSeek-R1 uses rule-based verification for tasks with objectively checkable answers and learned or preference-based rewards for open-ended tasks. Distinguish where RL with verifiable rewards can replace reward modeling from where reward modeling remains useful.
- Reasoning transfer through distillation. Describe how DeepSeek-R1-Distill models are trained and what the student models learn from the teacher.
- Complementary roles and limits. Distinguish what RL and distillation contribute, including the limit on what distillation can transfer.
- Purpose of the minimal code. Explain why the chapter code is deliberately small: it makes the core GRPO pieces concrete without the infrastructure required by a production trainer.
- Minimal math verifier and limits. Explain how the toy verifier checks an answer-only math completion and why it is not suitable for a production training run.
- Group-relative reward advantages. Explain how rewards for G completions per prompt are normalized across the group dimension, and describe the resulting signals for the example [1, 0, 1, 0].
- Policy roles in the loss. Describe how the loss brings together the current policy, the old policy that generated the completions, the frozen reference policy, and the advantages. Explain that a language-model trainer uses log probabilities for each generated token, stores old log probabilities during rollout, and computes reference log probabilities with the frozen model.
- GRPO training-step workflow. Trace the training step from sampling a group of completions through verification, group-relative advantages, policy scoring, GRPO loss, and the optimizer step.
- Group-relative rewards and updates. Trace how four sampled completions with rewards [1, 0, 1, 0] receive positive or negative advantages, and describe the resulting direction of the policy update.
- Final-answer feedback and scale. Explain what the verifier’s final-answer rewards do not tell GRPO about incorrect reasoning, and why repeated verifiable tasks can nevertheless favor productive reasoning paths.
- Completion advantage across tokens. Explain how a verifier’s score for a whole answer affects the generated tokens in that completion, and why this imperfect credit-assignment signal can still be useful.
- Constraints on token updates. Distinguish the direction illustrated by the simplified policy-gradient equation from the training rule described by the full GRPO objective.
- Prompt-local reward comparisons. Explain how sampling multiple completions for one prompt lets GRPO compare which responses are better or worse for that same input, rather than relying on raw rewards alone.
- Prompt-aware baseline advantage. Explain why comparing rewards within a prompt is useful when prompts vary in difficulty, and contrast this with a global baseline.
- Group size compute tradeoff. Describe how group size affects the estimate of a prompt's reward distribution and the rollout-compute budget.
- Generation and log probability scoring. Trace how a real trainer replaces the toy generation and scoring placeholders, from sampling completions to obtaining token-level log probabilities. Explain why it records probabilities under the policy that generated the rollout.
- Algorithm versus production infrastructure. Distinguish the toy code’s unchanged algorithmic flow from the scale and infrastructure needed for a real LLM trainer.
- Response format and reward reliability. Explain how ambiguous model outputs can make verifier rewards noisy even when a response is correct, and how structured final-answer formats or strict code-output protocols reduce that risk.
- Beyond Average Reward Monitoring. Explain why average reward alone is insufficient for monitoring a GRPO/RLVR run, and describe the other behaviors and metrics the section recommends tracking.
- Advantage Computation Code Change. Explain that a PPO-to-GRPO conversion replaces value-model-based advantages with advantages normalized from rewards within a sampled group, while retaining the old-policy log probabilities, probability ratio, clipped objective, and reference KL.
- Group Sampling Requirement. Explain why GRPO needs multiple completions per prompt and why removing the value model does not eliminate the rest of the training work.
- Post-training hook sequence. Trace how the verifier, group-advantage function, GRPO loss, and training-step skeleton fit together in a reasoning-oriented training stage.
- Discovery and transfer roles. Distinguish reinforcement learning’s role in creating or strengthening a reasoning teacher from distillation’s role in transferring that behavior to smaller students.
- Verifiable tasks and response protocol. Explain how to choose checkable tasks and define an answer format that a verifier can reliably score.
- Grouped samples and policy records. Describe what to generate and retain for each prompt, and explain the frozen reference policy's role.
- Rewards and group learning signal. Explain how completions become group-relative advantages and what to watch for when rewards provide little signal.
- Masked loss and training signals. Describe which token probabilities enter the GRPO loss and how training metrics can warn of problems.
- Held-out evaluation and system reliability. Justify evaluating beyond training reward and describe why a realistic trainer needs more than the short algorithm.
- Verifiable reward evidence and limits. Explain what R1-Zero supports about reinforcement learning with verifiable rewards and why that result does not extend to every alignment task.
- R1-Zero and R1 supervision. Distinguish what R1-Zero shows about supervised chain-of-thought from the supervised stages used to make DeepSeek-R1 usable.
- Response and padding masks. Explain which tokens should contribute to the training loss and which should be excluded.
- Verifier reliability and reward hacking. Explain how unreliable answer extraction or an incomplete checker can distort rewards, and describe the response-format safeguard given here.
What it builds on
- The training pipeline: Bringing the model to life (chapter 6) · The training pipeline produces the base model that is subsequently post-trained.
Section by section
Introduction
Introduces reinforcement-learning post-training as a way to turn a text-predicting base model toward rewarded reasoning behavior, using DeepSeek-R1 as the example.
- Reward-Based Post-Training: Explain how reinforcement-learning post-training differs from training a base model to predict text: the model tries complete responses, receives rewards, and is updated to make rewarded behavior more likely.
Why it matters: This distinction explains why the chapter adds post-training to the base model built earlier. - DeepSeek-R1 Reasoning Result: Describe the chapter’s claim that training a large base model with GRPO and verifiable rewards on objectively checkable tasks led it to produce longer, more self-corrective solutions, rather than simply memorizing human-written reasoning examples.
Why it matters: It connects the chapter’s reward-based training idea to the reasoning behavior it aims to explain.
Key terms: post-training, reinforcement learning, reward, base model, DeepSeek-R1, GRPO, verifiable rewards, long-form reasoning, objectively checkable tasks
7.1 The reinforcement learning framework
Introduces reinforcement learning as learning from rewarded interactions and explains why delayed feedback can help train language models for reasoning.
- Agent interaction and reward: Describe how an agent observes a state, takes an action, receives a reward, and observes a new state, learning over repeated interactions which actions tend to earn higher rewards.
Why it matters: This interaction pattern is the foundation for the chapter’s later discussion of rewards and model updates. - Feedback versus target answers: Distinguish supervised learning’s target answers from reinforcement learning’s potentially delayed feedback about whether behavior was useful.
Why it matters: The distinction explains why rewarding an outcome is different from providing a model with steps to imitate. - Final answer rewards for reasoning: Explain why rewarding a correct final answer can be useful for math reasoning when the desired sequence of intermediate tokens is unknown.
Why it matters: It connects delayed rewards to the chapter’s motivation for using reinforcement learning in reasoning tasks.
Key terms: agent, environment, state, action, reward, supervised learning, target answer, reinforcement learning, delayed feedback, final answer
7.1.1 The agent-environment interface
Uses a cleaning robot to explain the agent–environment loop and how a learned policy favors actions with higher long-term reward.
- Agent–environment interaction loop: Trace how the cleaning robot observes the room, chooses an action, and receives a reward after the room changes, then repeats the cycle.
Why it matters: This loop is the framework the chapter later translates into language-model terms. - Policy and reward-driven training: Explain what the robot’s policy does and how training changes a neural-network policy in relation to long-term reward.
Why it matters: It connects the robot’s choices to what reinforcement-learning training changes.
Key terms: agent, environment, state, action, reward, policy, model parameters, long-term reward
7.1.2 Mapping reinforcement learning to language generation
This section maps token-by-token language generation onto reinforcement learning and explains why rollouts sample tokens rather than always choosing the most likely one.
- Language Generation as RL: Trace how a prompt and generated tokens form states, how each sampled next token acts as an action, and when the completed response receives a reward.
Why it matters: This mapping makes it possible to discuss an LLM’s responses using reinforcement-learning concepts. - Sampling During RL Rollouts: Explain why reinforcement-learning rollouts sample from the model’s probability distribution instead of always selecting the most likely token.
Why it matters: Exploration gives training a chance to encounter response paths that a fixed most-likely choice might miss.
Key terms: state, action, transition, reward function, context, sampled, reinforcement-learning rollouts, probability distribution, greedy decoding
7.1.3 Sparse rewards and credit assignment
Explains why delayed rewards make token-level credit assignment difficult in language-model reinforcement learning and how negative learning signals can arise from advantages.
- Sparse Rewards and Credit Assignment: Explain that a verifier may reward a completed answer without scoring each token, so the reward does not reveal which token or mistake caused the outcome. Describe how policy-gradient methods respond by updating the probabilities of all sampled actions in proportion to that outcome.
Why it matters: This explains why training from a final-answer score is less direct than receiving feedback on each token. - Negative Advantages from Baselines: Explain how a zero verifier reward can produce a negative advantage when it falls below the baseline for that prompt, pushing the model away from the sampled behavior. Distinguish this from a reward function that assigns an explicitly negative value.
Why it matters: It clarifies how even a verifier that awards only 0 or 1 can discourage a sampled response.
Key terms: sparse rewards, credit-assignment problem, policy-gradient methods, sampled actions, raw verifier rewards, baseline, advantage, negative learning signals
7.2 Policy-gradient methods: Updating the LLM with rewards
Introduces how policy-gradient methods use rewards to change an LLM’s token probabilities.
- Reward-driven token probabilities: Explain how a policy-gradient method relates a rewarded rollout to changes in an LLM’s token probabilities, identifying the policy and its parameters.
Why it matters: This connects a reward for a generated response to the probabilities the model will assign to future tokens.
Key terms: policy-gradient method, policy, policy parameters, rollout, reward
7.2.1 Sampling actions instead of taking argmax
Explains why LLM reinforcement-learning rollouts sample tokens rather than always choosing the most likely token, and how sampled tokens form a rewarded response trajectory.
- Sampling for exploration: Explain why a rollout samples from the next-token probabilities instead of always choosing the highest-probability token.
Why it matters: Exploration gives training a chance to evaluate choices the model would not otherwise make. - Sampled response trajectories: Describe how sampled tokens make up a response trajectory and what probabilities the policy-gradient update changes.
Why it matters: This connects a reward for a whole response to updates of the token choices that produced it.
Key terms: rollout, stochasticity, exploration, argmax, log probability, sampled action, state, trajectory, reward
7.2.2 The basic policy-gradient update
This section explains how a basic policy-gradient update turns reward for sampled tokens into a change in the LLM’s parameters.
- Reward-weighted token updates: Explain how the log-probability gradient and reward affect the probability of a sampled token, and state that this simplified update uses raw reward without a baseline.
Why it matters: It connects a reward for a generated response to an update of the probabilities of tokens the model actually sampled.
Key terms: policy-gradient update, log probability, reward R, policy, sampled action, baseline
7.2.3 Why raw rewards are too noisy
This section explains why prompt-dependent baselines make reward-based LLM updates less noisy.
- Prompt-relative response quality: Explain why a raw reward may not show how good a completion is relative to other completions for the same prompt.
Why it matters: It explains why training on raw scores alone can give a misleading signal. - Advantages and baseline stability: Describe how an advantage compares an outcome with a baseline, and why using a baseline can make training updates less noisy.
Why it matters: It connects relative performance to more stable reward-based training.
Key terms: raw reward, prompt, advantage, baseline, variance, Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO)
7.3 PPO: The standard practical baseline
Introduces PPO’s trust-region intuition and explains how clipping a new-to-old policy probability ratio helps stabilize updates.
- PPO update stability: Explain why PPO limits changes to a policy after collecting rollouts, and describe how its probability ratio and clipping help do so.
Why it matters: This explains how PPO tries to keep learning from a batch of rollouts stable.
Key terms: Proximal Policy Optimization, PPO, trust-region, probability ratio, old policy, new policy
7.3.1 PPO components for LLM training
Introduces the four components of PPO for LLM post-training and explains the distinct roles of the value model and frozen reference model.
- PPO training components: Describe the four components of PPO for LLM post-training: a trainable policy LLM, a reward signal, a value model, and a frozen reference model. Explain that the value model estimates expected reward to provide a baseline for advantage calculation.
Why it matters: Keeping these roles separate makes PPO’s training setup easier to reason about. - Reference model drift control: Explain why PPO keeps a frozen pre-RL reference model and uses a KL penalty when the trained policy moves too far from it.
Why it matters: A high reward alone does not ensure that the updated model remains a useful assistant.
Key terms: trainable policy model, reward signal, value model, reference model, advantage, KL control, frozen reference model, reference policy, KL penalty, drift
7.3.2 Where the value function comes from
Explains how PPO’s value model learns to predict returns from partial responses and why GRPO removes that model.
- Learned Value Function: Explain how the value model learns to predict the expected return from a partial response using states observed during rollouts and their eventual returns.
Why it matters: This explains where PPO’s estimate of a partial response’s likely outcome comes from. - Policy And Value Roles: Distinguish the policy’s token-selection role from the value model’s prediction role, and explain why GRPO removes the learned value model.
Why it matters: The distinction makes clear what GRPO simplifies without implying that it abandons a baseline.
Key terms: value function, value model, partial states, eventual returns, policy, θ, ϕ, GRPO, baseline
7.3.3 Why the reference model matters
This section explains why PPO-style LLM training constrains the trained policy with a reference model and why GRPO retains that constraint.
- Reference policy constraint: Explain why an imperfect reward function calls for a reference-policy constraint and how a token-level KL penalty discourages the trained policy from moving too far away.
Why it matters: It explains why improving the reward alone is not the whole training objective. - Reference model in GRPO: Distinguish the value model that GRPO removes from the reference model and its KL penalty, which GRPO retains.
Why it matters: It prevents confusing GRPO's value-model simplification with removal of the reference-policy constraint.
Key terms: reference model, reference policy, KL term, reward functions, GRPO, value model, reference-policy KL penalty
7.4 GRPO: DeepSeek's value model simplification
This section introduces GRPO’s replacement of a learned value model with prompt-specific comparisons among sampled completions, and notes how it handles reference-model KL.
- Group-relative reward baseline: Explain how GRPO samples multiple completions for one prompt, rewards each, and normalizes their rewards so that better-than-group completions have positive advantages and worse-than-group completions have negative advantages, without a learned value model.
Why it matters: Comparing answers to the same prompt explains how GRPO gets a useful baseline without training a critic. - Separate reference-model KL term: Distinguish GRPO’s separate reference-model KL loss term from the practice of folding KL into the reward signal, and explain why keeping it separate helps with tuning.
Why it matters: Keeping KL separate makes its influence on training easier to see and adjust.
Key terms: GRPO, value model, group of completions, baseline, advantages, reference-model KL, separate term in the loss, reward signal, k3 / Schulman estimator
7.4.1 Group-relative advantages
This section explains how GRPO compares completions for the same prompt to assign advantages, and how those advantages relate to token-level training.
- Group-relative advantage calculation: Explain how rewards from multiple completions of the same prompt are turned into advantages by comparing each reward with the group mean and accounting for the group's reward variation.
Why it matters: The optimizer uses relative advantages, not just the verifier's raw reward, to compare responses to the same prompt. - Advantages in token-level training: Describe what a negative advantage signals to the optimizer and how a completion-level advantage is commonly applied to generated tokens without including prompt or padding tokens in the loss.
Why it matters: It connects a score for an entire completion to updates made through its generated tokens.
Key terms: group-relative advantage, group mean, standard deviation, ε, completion, reward, negative advantage, sampled trajectory, completion-level advantage, generated tokens
7.4.2 The full GRPO objective
This section identifies the stabilizing terms in the full GRPO objective and explains why its PyTorch minimization loss is negated.
- Full GRPO objective components: Explain that GRPO uses a group-relative advantage while retaining PPO’s clipped policy ratio and a KL term against a frozen reference policy. Distinguish what each term does.
Why it matters: A group-relative advantage alone leaves out safeguards that the full objective retains. - Minimization loss sign: Explain why a PyTorch implementation usually negates the GRPO objective when using a minimization loss.
Why it matters: The sign determines whether optimization increases the intended objective.
Key terms: group-relative advantage, clipped ratio, old policy, KL term, frozen reference policy, minimization loss, negative objective
7.4.3 Why GRPO is cheaper than PPO
This section explains why GRPO is cheaper than PPO when both are compared at the same number of sampled completions.
- GRPO cost advantage over PPO: Explain that GRPO's cost advantage comes from eliminating the learned value model and its training path, not from sampling fewer completions.
Why it matters: Comparing equal sampling workloads isolates the source of GRPO's savings.
Key terms: GRPO, PPO, sampled completions, learned value model, value function
7.5 Reinforcement learning with verifiable rewards
Introduces reinforcement learning with verifiable rewards (RLVR), explaining how deterministic checks provide reward signals and where that approach cannot replace other alignment signals.
- Deterministic reward signals: Explain that RLVR assigns rewards using deterministic checkers instead of a learned reward model, and give an example of a check for math, code, or a symbolic task.
Why it matters: It explains why verifiable tasks can supply rewards at scale without judging each response by hand. - Limits of verifiable rewards: Distinguish tasks with objectively verifiable correctness from open-ended assistant tasks, and explain why the latter still need other alignment signals.
Why it matters: It prevents treating automatic correctness checks as a complete solution to assistant alignment.
Key terms: reinforcement learning with verifiable rewards, RLVR, deterministic checker, learned reward model, objectively verifiable, open-ended dialog helpfulness, preference data, human evaluation
7.5.1 Designing a verifier
This section explains how verifier design balances correctness with formatting flexibility and how its reward rules shape model behavior.
- Correctness and formatting balance: Explain how a verifier can reward correct answers without rejecting harmless differences in how they are written. Describe the toy math check and why a production math verifier may need more robust comparisons.
Why it matters: A brittle verifier can penalize a right answer simply because it is expressed differently. - Verifier rewards shape behavior: Explain how the checks included in a reward function can change the model's responses, using the correctness and format rewards described for DeepSeek-R1-Zero.
Why it matters: Reward rules influence not just whether the final answer is right, but how the model presents its response.
Key terms: verifier, gold answer, symbolic equivalence, unit normalization, structured answer extraction, reward function, rule-based rewards, correctness reward, format reward, reasoning/final-answer structure
7.5.2 Avoiding overstatement
This section limits the claim for reinforcement learning with verifiable rewards (RLVR) and explains when to use verifiers versus learned reward models.
- Limits of verifiable rewards: Distinguish tasks suited to deterministic verification from tasks that require preference judgments, and explain why reward models remain useful in a hybrid reward design.
Why it matters: It helps you choose a reward signal without assuming every desired behavior can be checked mechanically.
Key terms: RLVR, deterministic verifier, hybrid reward design, rule-based rewards, learned reward models, preference judgments
7.6 How reasoning emerges in DeepSeek-R1-Zero
Explains what DeepSeek-R1-Zero’s reinforcement-learning experiment shows about emerging reasoning behavior, and where that conclusion has limits.
- R1-Zero training setup: Describe how DeepSeek-R1-Zero starts from DeepSeek-V3-Base and applies large-scale reinforcement learning with rewards for verifiable reasoning tasks, without first training on supervised chain-of-thought examples.
Why it matters: This setup is what makes R1-Zero evidence that reasoning-like behavior can arise without supervised solution traces. - Emerging reasoning behavior: Explain how the model’s use of more tokens on difficult problems can provide room to try, check, and revise an approach, and what the reported 'aha moment' illustrates. Describe what the <think> tag denotes in that example.
Why it matters: The example makes the observed behavior more specific than simply saying that the model produced longer answers. - Limits of R1-Zero findings: Distinguish the finding that reasoning-like behavior can emerge under RL pressure on verifiable tasks from claims that any reward guarantees reasoning or that R1-Zero is a polished model.
Why it matters: Keeping the result and its limits separate prevents an observed training outcome from becoming an overgeneralized claim.
Key terms: DeepSeek-R1-Zero, DeepSeek-V3-Base, supervised chain-of-thought examples, verifiable reasoning tasks, aha moment, <think> tag, reasoning region, reasoning-like behavior, verifiable tasks, poor readability
7.7 DeepSeek-R1, R1-Zero, and Distill
Distinguishes the three related DeepSeek-R1 artifacts by how they are produced and what role each serves.
- DeepSeek-R1 model distinctions: Distinguish DeepSeek-R1-Zero, DeepSeek-R1, and DeepSeek-R1-Distill by their training approach and purpose.
Why it matters: Keeping the three names separate makes it easier to follow which training approach and model later sections discuss.
Key terms: DeepSeek-R1-Zero, DeepSeek-R1, DeepSeek-R1-Distill, verifiable rewards, cold-start supervised fine-tuning
7.7.1 From R1-Zero to R1
This subsection explains how DeepSeek-R1 develops from filtered R1-Zero outputs through supervised fine-tuning and reinforcement learning stages, rather than by simply continuing R1-Zero training.
- R1-Zero to R1 data flow: Distinguish the flow of training data from R1-Zero to R1 from a continuation of R1-Zero's parameters. Explain why the sampled reasoning traces need filtering and refinement.
Why it matters: This distinction prevents you from treating two different training pipelines as one continuous model run. - Cold start and reasoning RL: Describe how cold-start supervised examples establish a readable reasoning format before reasoning-oriented RL, and how that RL stage addresses language mixing.
Why it matters: The order explains how R1 starts heavy RL with a more readable reasoning style. - Filtered data and assistant behavior: Explain how rejection sampling and supervised fine-tuning use filtered outputs and broader instruction data to restore general assistant behavior.
Why it matters: Improving verifiable reasoning alone does not directly train the full range of assistant behavior. - Final hybrid reward signals: Distinguish the reward sources in R1's final RL stage for verifiable tasks and open-ended helpfulness and harmlessness tasks.
Why it matters: The hybrid reward explains why the final pipeline can train on both checkable answers and open-ended assistant qualities.
Key terms: R1-Zero, R1, data flow, parameter flow, reasoning traces, cold-start supervised data, chain-of-thought, reasoning-oriented RL, rule-based rewards, language-consistency reward
7.7.2 Why cold-start data matters
Explains why DeepSeek-R1 uses a small supervised cold-start dataset before reasoning RL to establish readable, structured responses and make RL training more effective.
- Cold-start and RL roles: Explain why DeepSeek-R1 uses curated supervised reasoning examples before its main RL stage, and how supervised fine-tuning and RL address different needs.
Why it matters: The cold start explains why reasoning ability alone was not enough for the intended assistant behavior. - Structured outputs and RL signal: Explain how a small cold-start dataset can reduce wasted RL exploration and make verifier feedback cleaner.
Why it matters: A usable response format helps training feedback reflect the answer rather than avoidable parsing or format failures.
Key terms: cold-start, supervised fine-tuning, reinforcement learning, R1-Zero, cold-start dataset, final answer region, verifier, RL signal, rollouts
7.7.3 Rejection sampling as data construction
Explains how the R1 pipeline turns selected outputs from an RL-trained model into supervised data for a more balanced checkpoint.
- Rejection Sampling Data Construction: Describe how rejection sampling generates supervised training data by sampling candidate responses, retaining those that meet correctness and quality criteria, and discarding the rest.
Why it matters: It explains how the pipeline obtains training examples from a model's strongest outputs. - Filtering Criteria and Pipeline Flow: Explain why the pipeline filters an RL-trained model's outputs and how those outputs contribute to R1 training rather than simply transferring weights from the pure-RL experiment.
Why it matters: The distinction clarifies that R1 benefits from RL-generated data without being just a copy of the pure-RL model.
Key terms: rejection sampling, candidate responses, supervised training data, RL-trained model, rule-based verifier, data flow, balanced checkpoint
7.7.4 Restoring general assistant behavior
This section explains why verifiable-reward training alone may narrow an assistant’s behavior and how DeepSeek-R1 uses broader supervised data to preserve general usefulness.
- Limits of narrow rewards: Explain why optimizing heavily for math and code verifiers can improve verifiable reasoning without producing a polished general assistant.
Why it matters: Success on verifiable tasks does not by itself show that a model will handle ordinary requests well. - Broad supervised data mix: Describe how DeepSeek-R1 uses mixed supervised data after rejection sampling to help preserve general assistant behavior, and distinguish that step from relying on GRPO and verifiable rewards alone.
Why it matters: A capable general assistant needs training signals for everyday tasks as well as verifiable reasoning.
Key terms: verifiable reasoning, general assistant, narrow reward, mixed supervised data, reasoning examples, non-reasoning instruction data, pipeline
7.7.5 The final hybrid reward stage
Explains why DeepSeek-R1’s final reinforcement-learning stage combines verifiable and preference-based rewards, and how those rewards complement supervised training.
- Task-matched hybrid rewards: Explain why DeepSeek-R1 uses rule-based verification for tasks with objectively checkable answers and learned or preference-based rewards for open-ended tasks. Distinguish where RL with verifiable rewards can replace reward modeling from where reward modeling remains useful.
Why it matters: Choosing a reward that fits the task is the central design decision in this final RL stage. - Complementary training contributions: Describe the distinct contributions the section attributes to verifiable rewards, supervised data, and preference-oriented rewards in the final R1 model.
Why it matters: The final model’s capabilities are not attributed to verifiable-reward RL alone.
Key terms: hybrid reward, RLVR, rule-based checker, learned reward model, preference-based signal, verifiable rewards, supervised data, preference-oriented rewards
7.7.6 Distillation into smaller models
This section explains how DeepSeek-R1 transfers reasoning behavior to smaller models through supervised fine-tuning, and why that approach complements rather than replaces RL.
- Reasoning transfer through distillation: Describe how DeepSeek-R1-Distill models are trained and what the student models learn from the teacher.
Why it matters: It explains how a large model’s reasoning behavior can reach smaller models without training each one through large-scale RL. - Distillation training economics: Explain why distillation can be a practical way to train smaller reasoning models and what comparison the section reports.
Why it matters: The training-cost difference helps explain why one strong teacher can support multiple smaller deployments. - Complementary roles and limits: Distinguish what RL and distillation contribute, including the limit on what distillation can transfer.
Why it matters: It prevents treating the cheaper transfer method as a substitute for discovering or improving the teacher’s behavior.
Key terms: DeepSeek-R1-Distill, supervised dataset, reasoning traces, fine-tuned, RL, supervised fine-tuning, distillation, reasoning benchmarks, teacher, student
7.8 Minimal chapter code
This section explains why the chapter uses a small GRPO code example rather than a production RL trainer.
- Purpose of the minimal code: Explain why the chapter code is deliberately small: it makes the core GRPO pieces concrete without the infrastructure required by a production trainer.
Why it matters: It sets the right expectations for what the example code does and does not provide.
Key terms: verifier rewards, group-relative advantages, clipped GRPO loss, reference-policy KL, training-step skeleton
7.8.1 A minimal verifier
This subsection describes a toy math-answer verifier and explains why it is insufficient for production training.
- Minimal math verifier and limits: Explain how the toy verifier checks an answer-only math completion and why it is not suitable for a production training run.
Why it matters: The reward is only as reliable as the check used to judge the completion.
Key terms: deterministic verifier, gold answer, RLVR
7.8.2 Computing group-relative advantages
Shows how GRPO turns rewards for multiple completions of each prompt into group-relative advantage signals.
- Group-relative reward advantages: Explain how rewards for G completions per prompt are normalized across the group dimension, and describe the resulting signals for the example [1, 0, 1, 0].
Why it matters: The sign of each advantage indicates whether that completion should be reinforced or penalized relative to others for the same prompt.
Key terms: group-relative baseline, group dimension, normalized advantages, binary verifier rewards
7.8.3 The clipped GRPO loss
This section identifies the inputs to the clipped GRPO loss and explains why the illustrative listing is not a production-ready trainer.
- Policy roles in the loss: Describe how the loss brings together the current policy, the old policy that generated the completions, the frozen reference policy, and the advantages. Explain that a language-model trainer uses log probabilities for each generated token, stores old log probabilities during rollout, and computes reference log probabilities with the frozen model.
Why it matters: Keeping the three policy roles separate makes the training loss easier to interpret and implement. - Illustrative loss limitations: Explain why the listing is only a structural illustration: a real trainer must exclude prompt tokens from the policy loss, ignore padding tokens, and may use a more carefully chosen KL estimate, such as the low-variance Schulman (k3) estimator used in DeepSeek-style GRPO training.
Why it matters: The example shows the shape of the loss without covering safeguards needed in a real trainer.
Key terms: current policy, old policy, frozen reference policy, advantages, log probabilities, rollout, attention masks, response masks, policy loss, padding tokens
7.8.4 One training-step skeleton
Shows how one GRPO training step connects sampling, verification, group-relative advantages, policy scoring, loss computation, and an optimizer update.
- GRPO training-step workflow: Trace the training step from sampling a group of completions through verification, group-relative advantages, policy scoring, GRPO loss, and the optimizer step.
Why it matters: This sequence shows where each previously introduced GRPO component fits in an update. - Rollout and scoring placeholders: Describe what sample_group and sequence_logprobs must provide when the skeleton is connected to a real LLM.
Why it matters: The placeholder contracts identify what a real rollout and scoring implementation must pass into the GRPO step.
Key terms: group of completions, verifiable reward, group-relative advantages, GRPO loss, optimizer step, sample_group, sequence_logprobs, old-policy token log probabilities, current policy, frozen reference model
7.9 A worked GRPO example
A four-completion arithmetic example shows how GRPO turns group-relative rewards into a policy update and why final-answer rewards require many training examples.
- Group-relative rewards and updates: Trace how four sampled completions with rewards [1, 0, 1, 0] receive positive or negative advantages, and describe the resulting direction of the policy update.
Why it matters: The example connects a group of verifier scores to the direction in which GRPO changes the policy. - Final-answer feedback and scale: Explain what the verifier’s final-answer rewards do not tell GRPO about incorrect reasoning, and why repeated verifiable tasks can nevertheless favor productive reasoning paths.
Why it matters: This explains both the limitation of sparse final-answer feedback and why it can still help at scale.
Key terms: raw rewards, group mean, positive advantages, negative advantages, clipping, KL control, verifier, final answers, RLVR, reasoning patterns
7.9.1 What happens to each token?
Explains how a whole-answer score affects each generated token in a GRPO update, and why the full objective differs from a simplified policy-gradient equation.
- Completion advantage across tokens: Explain how a verifier’s score for a whole answer affects the generated tokens in that completion, and why this imperfect credit-assignment signal can still be useful.
Why it matters: It clarifies how answer-level feedback can train a token-generating model without assigning a separate reward to each word. - Constraints on token updates: Distinguish the direction illustrated by the simplified policy-gradient equation from the training rule described by the full GRPO objective.
Why it matters: A positive advantage alone does not determine how large the final policy change will be.
Key terms: verifier, completion-level advantage, token log probabilities, credit-assignment signal, policy-gradient equation, GRPO objective, clipping, reference-policy KL, group-relative advantages
7.9.2 Why multiple samples for the same prompt matter
Explains why GRPO compares multiple completions for each prompt and how group size trades off comparison quality against rollout cost.
- Prompt-local reward comparisons: Explain how sampling multiple completions for one prompt lets GRPO compare which responses are better or worse for that same input, rather than relying on raw rewards alone.
Why it matters: The comparison group is central to why GRPO samples several answers to one prompt. - Prompt-aware baseline advantage: Explain why comparing rewards within a prompt is useful when prompts vary in difficulty, and contrast this with a global baseline.
Why it matters: It explains why a reward should be interpreted relative to the difficulty of its prompt. - Group size compute tradeoff: Describe how group size affects the estimate of a prompt's reward distribution and the rollout-compute budget.
Why it matters: Practical group size balances the quality of each comparison against how many prompts training can cover. - GRPO engineering compromise: Describe the specific model and compute choices the section identifies as part of DeepSeek's use of GRPO.
Why it matters: It connects the benefit of local comparisons to the resources GRPO does and does not use.
Key terms: GRPO, group structure, local comparison, completions, global baseline, group-relative baseline, advantage, prompt-aware, group size, prompt-specific reward distribution
7.10 From toy code to a real LLM trainer
Maps the chapter’s minimal GRPO code to a real LLM trainer, identifying the generation, scoring, masking, and infrastructure it leaves out.
- Generation and log probability scoring: Trace how a real trainer replaces the toy generation and scoring placeholders, from sampling completions to obtaining token-level log probabilities. Explain why it records probabilities under the policy that generated the rollout.
Why it matters: These details connect sampled completions to the policy-gradient update without comparing the wrong probabilities or token positions. - Variable length loss masking: Explain how a real trainer batches completions of different lengths without training on prompt or padding tokens.
Why it matters: Correct masking keeps the update focused on actual generated responses. - Algorithm versus production infrastructure: Distinguish the toy code’s unchanged algorithmic flow from the scale and infrastructure needed for a real LLM trainer.
Why it matters: This lets you interpret the toy trainer as an algorithmic example without mistaking it for a production-ready system.
Key terms: sample_group, sequence_logprobs, old-policy log probability, PPO/GRPO ratio, padding, masks, loss, algorithmic data flow, distributed execution, rollout generation
7.10.1 Response format and verifier reliability
Explains why reliable RLVR rewards depend on response formats that verifiers can parse consistently.
- Response format and reward reliability: Explain how ambiguous model outputs can make verifier rewards noisy even when a response is correct, and how structured final-answer formats or strict code-output protocols reduce that risk.
Why it matters: A training reward is only useful if the verifier evaluates the response the model actually intended to submit.
Key terms: RLVR, verifier, structured final-answer format, format rewards, strict output protocols
7.10.2 trainiMonitoring behavior
This section explains how to monitor GRPO/RLVR training so rising reward or longer responses are not mistaken for better reasoning.
- Beyond Average Reward Monitoring: Explain why average reward alone is insufficient for monitoring a GRPO/RLVR run, and describe the other behaviors and metrics the section recommends tracking.
Why it matters: A reward curve alone can conceal a training run that is getting worse in important ways. - Correctness Versus Verbosity: Explain how task-specific evaluation and correctness checks help distinguish useful reasoning gains from longer answers or exploitable visible rewards.
Why it matters: It keeps apparent reasoning gains tied to solving more problems, not merely producing more text or exploiting an evaluation.
Key terms: average reward, pass rate, response length, KL divergence, invalid-format rate, duplicate-output rate, hidden-test performance, visible-test reward, held-out set, robust answer checking
7.10.3 What changes from PPO to GRPO in code?
Explains which parts of a PPO trainer remain in GRPO, which code paths change, and why the rollout structure must change too.
- Advantage Computation Code Change: Explain that a PPO-to-GRPO conversion replaces value-model-based advantages with advantages normalized from rewards within a sampled group, while retaining the old-policy log probabilities, probability ratio, clipped objective, and reference KL.
Why it matters: This identifies what to change in an existing trainer without mistakenly rewriting everything. - Group Sampling Requirement: Explain why GRPO needs multiple completions per prompt and why removing the value model does not eliminate the rest of the training work.
Why it matters: The simpler model stack only works if the trainer supplies the groups of responses its advantage calculation requires.
Key terms: advantage, value model, group normalization, clipped objective, reference KL, multiple completions per prompt, group baseline, rollout structure, KL control
7.11 What does this chapter add to the book?
This section connects the chapter’s post-training hooks to the distinction between using reinforcement learning to strengthen reasoning and using distillation to transfer it.
- Post-training hook sequence: Trace how the verifier, group-advantage function, GRPO loss, and training-step skeleton fit together in a reasoning-oriented training stage.
Why it matters: This sequence shows how the chapter’s small code components connect rewards to a policy update. - Discovery and transfer roles: Distinguish reinforcement learning’s role in creating or strengthening a reasoning teacher from distillation’s role in transferring that behavior to smaller students.
Why it matters: The distinction explains why a reasoning teacher can provide training data for a smaller model.
Key terms: verifier function, group-advantage function, GRPO loss, training-step skeleton, reinforcement learning, distillation, teacher, students, supervised fine-tuning, DeepSeek-R1-Zero
7.12 A practical implementation checklist
This section turns a minimal GRPO script into a checklist for building a reliable reinforcement-learning experiment.
- Verifiable tasks and response protocol: Explain how to choose checkable tasks and define an answer format that a verifier can reliably score.
Why it matters: If answers cannot be checked reliably, the training reward may not reflect task success. - Grouped samples and policy records: Describe what to generate and retain for each prompt, and explain the frozen reference policy's role.
Why it matters: Grouped completions and the right policy records are prerequisites for computing the intended GRPO update. - Rewards and group learning signal: Explain how completions become group-relative advantages and what to watch for when rewards provide little signal.
Why it matters: A reward pipeline is useful only when its scores distinguish responses and its failures can be diagnosed. - Masked loss and training signals: Describe which token probabilities enter the GRPO loss and how training metrics can warn of problems.
Why it matters: Correct masking keeps the update focused on generated answers, while monitoring helps catch misleading apparent progress. - Held-out evaluation and system reliability: Justify evaluating beyond training reward and describe why a realistic trainer needs more than the short algorithm.
Why it matters: A working training loop is not the same as evidence that the model generalizes or that the experiment is reliable.
Key terms: task distribution, gold answers, response protocol, verifier, reference policy, KL anchor, prompt group, old-policy log probabilities, response masks, reward
7.13 Reading DeepSeek-R1 claims carefully
Separates evidence-backed DeepSeek-R1 claims from overclaims about verifiable rewards, GRPO, supervision, and distillation.
- Verifiable reward evidence and limits: Explain what R1-Zero supports about reinforcement learning with verifiable rewards and why that result does not extend to every alignment task.
Why it matters: It keeps evidence of reasoning gains from becoming a claim that verifiable rewards solve all alignment. - GRPO value model simplification: Describe the supported efficiency claim about GRPO and what remains in its RL update.
Why it matters: It identifies the specific simplification behind the claim that GRPO training is cheaper. - R1-Zero and R1 supervision: Distinguish what R1-Zero shows about supervised chain-of-thought from the supervised stages used to make DeepSeek-R1 usable.
Why it matters: It prevents a result from a pure RL experiment from being mistaken for a complete assistant-training recipe. - Reasoning transfer through distillation: State the supported claim about transferring reasoning behavior to smaller models through distillation.
Why it matters: It distinguishes transferring a capable model's behavior from running the same RL process on every smaller model.
Key terms: GRPO, verifiable rewards, DeepSeek-R1-Zero, RLVR, value model, learned value baseline, group-relative reward normalization, critic, clipping, R1-Zero
7.14 Common implementation pitfalls
Identifies implementation pitfalls that matter when scaling a compact GRPO training example.
- Response and padding masks: Explain which tokens should contribute to the training loss and which should be excluded.
Why it matters: Masking keeps the update focused on the response the model generated. - Verifier reliability and reward hacking: Explain how unreliable answer extraction or an incomplete checker can distort rewards, and describe the response-format safeguard given here.
Why it matters: The reward signal is only as trustworthy as the checker and its interpretation of the response. - KL pressure balance: Describe the risks of setting KL-control beta too low or too high and what practical systems may monitor.
Why it matters: KL control must balance useful learning against excessive policy drift. - Group size and compute allocation: Explain the tradeoff in choosing group size and name the competing uses of the compute budget identified here.
Why it matters: A better baseline is not free: it competes with other demands on training compute.
Key terms: response masking, prompt tokens, padding tokens, loss, reward parsing, verifier, RLVR, reward hacking, final answer, KL control
7.15 Summary
Where this is used later
- Looking ahead (chapter 8) · GRPO as a reinforcement-learning approach.
- The 800K training dataset: Rejection sampling at scale (chapter 8) · Connects rejection sampling here to the later discussion of constructing a large distillation dataset.
- From logits to language: a paradigm shift (chapter 8) · Distinguishes training on the teacher’s generated reasoning from repeating the teacher’s RL training.
- (chapter intro) (chapter 8) · Introduces distillation into smaller models.
- The critical comparison: Distillation versus direct RL (chapter 8) · Provides the basis for comparing distillation with direct RL on smaller models.
- The knowledge frontier (chapter 8) · Develops the limit on what a student can gain from its teacher.
- DeepSeek-R1’s distillation recipe (chapter 8) · The DeepSeek-R1 distillation recipe builds on the distinction between developing teacher behavior and transferring it to students.
Concepts explained in this chapter
- Group Relative Policy OptimizationGRPO, or Group Relative Policy Optimization, is a reinforcement-learning algorithm that estimates relative advantages among grouped sampled responses and optimizes a policy with a clipped objective and regularization.
- reward signalA scalar reward is a single numerical value representing the quality or desirability of an agent's completed response or action.
- advantage functionAn advantage is a reinforcement-learning signal representing how much better or worse an action's outcome is than a reference baseline.
- policyA policy is a decision rule that assigns action probabilities or actions to situations encountered by a reinforcement-learning agent.
- reference modelA reference policy is a fixed or separately maintained policy whose output probabilities provide a comparison distribution during policy optimization.
- value functionA value function estimates the expected return associated with a state, action, or policy in reinforcement learning.
- log probabilityA log probability is the logarithm of the probability assigned to an event, such as a generated token under a language-model policy.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent improves its behavior by optimizing rewards received from interactions or evaluations.
- reward modelA reward model is a learned model that scores outputs according to preference-based or otherwise non-mechanical judgments of desired behavior.
- policy gradient methodA policy-gradient loss is an optimization objective that weights policy log probabilities by feedback-derived advantages to update a policy.
- verifiable rewardA verifiable reward is a training reward whose correctness can be determined by an executable checker, test, or other objective procedure.
- baselineA baseline model is a reference model trained under a simpler or comparison-only procedure without teacher guidance.
- verifierA deterministic checker is a procedure that evaluates an output against fixed criteria and produces the same result for the same input.
- distillationKnowledge distillation is a model-training technique in which a student model learns to reproduce information or behavior provided by a teacher model.
- KL regularizationKL control is the regulation of policy updates by penalizing divergence between the learned policy and a reference policy.
- fine-tuningFine-tuning is continued training of a pretrained model on a targeted dataset to adapt its behavior or capabilities.
- DeepSeek-R1-ZeroDeepSeek-R1-Zero is a reasoning-focused language model trained to develop reasoning behavior through reinforcement learning without conventional supervised fine-tuning as its initial reasoning-training stage.
- actionAn action is a choice made by an agent at a decision point that can change the state of an environment.
- reinforcement learning with verifiable rewardsReinforcement learning with verifiable rewards is reinforcement learning in which a deterministic procedure evaluates outputs and supplies the reward.
- chain-of-thoughtReasoning behavior is a model's capability to carry out or express multi-step inference and problem-solving strategies.
- Proximal Policy OptimizationProximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that constrains policy updates using a clipped or otherwise limited probability ratio.
- stateA state is the information describing the current situation from which a reinforcement-learning policy selects an action.
- cold-start datasetA cold-start dataset is a relatively small initial dataset of curated examples used to establish a model's behavior before later optimization.
- Kullback–Leibler divergenceKL divergence is a measure of the discrepancy between two probability distributions that quantifies information lost when one distribution approximates another.
- pipelineA training pipeline is an organized sequence of data preparation, model computation, and optimization operations for training a machine-learning model.
- DeepSeek-R1DeepSeek-R1 is a reasoning-focused large language model designed to achieve strong performance on demanding reasoning tasks.
- rejection samplingRejection sampling is a data curation procedure that generates multiple candidates and retains only those meeting specified correctness and quality criteria.
- generationGeneration is the process of producing a sequence of tokens from a language model, typically by sampling or selecting successive outputs.
- post-trainingPost-training is the stage of adapting a pretrained model through additional optimization, data generation, or alignment procedures.
- student modelA distilled model is a student model trained to reproduce capabilities or behavior transferred from a larger teacher model.
- teacher modelA teacher model is a model whose outputs or learned behavior provide supervision for training another model.
- data explorationExploration is the process of trying varied actions or strategies to discover which ones produce desirable outcomes during learning.
- evaluation datasetA validation set is a held-out subset of data used to measure how well a model generalizes during development.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
