liveBook · Manning

Chapter 7: Reinforcement learning: From policy gradients to GRPO

What you will learn in chapter 7 of Build a DeepSeek Model (From Scratch)

Introduces reinforcement-learning post-training as a way to turn a text-predicting base model toward rewarded reasoning behavior, using DeepSeek-R1 as the example.

After this chapter you will be able to

What it builds on

Section by section

Introduction

Introduces reinforcement-learning post-training as a way to turn a text-predicting base model toward rewarded reasoning behavior, using DeepSeek-R1 as the example.

Key terms: post-training, reinforcement learning, reward, base model, DeepSeek-R1, GRPO, verifiable rewards, long-form reasoning, objectively checkable tasks

7.1 The reinforcement learning framework

Introduces reinforcement learning as learning from rewarded interactions and explains why delayed feedback can help train language models for reasoning.

Key terms: agent, environment, state, action, reward, supervised learning, target answer, reinforcement learning, delayed feedback, final answer

7.1.1 The agent-environment interface

Uses a cleaning robot to explain the agent–environment loop and how a learned policy favors actions with higher long-term reward.

Key terms: agent, environment, state, action, reward, policy, model parameters, long-term reward

7.1.2 Mapping reinforcement learning to language generation

This section maps token-by-token language generation onto reinforcement learning and explains why rollouts sample tokens rather than always choosing the most likely one.

Key terms: state, action, transition, reward function, context, sampled, reinforcement-learning rollouts, probability distribution, greedy decoding

7.1.3 Sparse rewards and credit assignment

Explains why delayed rewards make token-level credit assignment difficult in language-model reinforcement learning and how negative learning signals can arise from advantages.

Key terms: sparse rewards, credit-assignment problem, policy-gradient methods, sampled actions, raw verifier rewards, baseline, advantage, negative learning signals

7.2 Policy-gradient methods: Updating the LLM with rewards

Introduces how policy-gradient methods use rewards to change an LLM’s token probabilities.

Key terms: policy-gradient method, policy, policy parameters, rollout, reward

7.2.1 Sampling actions instead of taking argmax

Explains why LLM reinforcement-learning rollouts sample tokens rather than always choosing the most likely token, and how sampled tokens form a rewarded response trajectory.

Key terms: rollout, stochasticity, exploration, argmax, log probability, sampled action, state, trajectory, reward

7.2.2 The basic policy-gradient update

This section explains how a basic policy-gradient update turns reward for sampled tokens into a change in the LLM’s parameters.

Key terms: policy-gradient update, log probability, reward R, policy, sampled action, baseline

7.2.3 Why raw rewards are too noisy

This section explains why prompt-dependent baselines make reward-based LLM updates less noisy.

Key terms: raw reward, prompt, advantage, baseline, variance, Proximal Policy Optimization (PPO), Group Relative Policy Optimization (GRPO)

7.3 PPO: The standard practical baseline

Introduces PPO’s trust-region intuition and explains how clipping a new-to-old policy probability ratio helps stabilize updates.

Key terms: Proximal Policy Optimization, PPO, trust-region, probability ratio, old policy, new policy

7.3.1 PPO components for LLM training

Introduces the four components of PPO for LLM post-training and explains the distinct roles of the value model and frozen reference model.

Key terms: trainable policy model, reward signal, value model, reference model, advantage, KL control, frozen reference model, reference policy, KL penalty, drift

7.3.2 Where the value function comes from

Explains how PPO’s value model learns to predict returns from partial responses and why GRPO removes that model.

Key terms: value function, value model, partial states, eventual returns, policy, θ, ϕ, GRPO, baseline

7.3.3 Why the reference model matters

This section explains why PPO-style LLM training constrains the trained policy with a reference model and why GRPO retains that constraint.

Key terms: reference model, reference policy, KL term, reward functions, GRPO, value model, reference-policy KL penalty

7.4 GRPO: DeepSeek's value model simplification

This section introduces GRPO’s replacement of a learned value model with prompt-specific comparisons among sampled completions, and notes how it handles reference-model KL.

Key terms: GRPO, value model, group of completions, baseline, advantages, reference-model KL, separate term in the loss, reward signal, k3 / Schulman estimator

7.4.1 Group-relative advantages

This section explains how GRPO compares completions for the same prompt to assign advantages, and how those advantages relate to token-level training.

Key terms: group-relative advantage, group mean, standard deviation, ε, completion, reward, negative advantage, sampled trajectory, completion-level advantage, generated tokens

7.4.2 The full GRPO objective

This section identifies the stabilizing terms in the full GRPO objective and explains why its PyTorch minimization loss is negated.

Key terms: group-relative advantage, clipped ratio, old policy, KL term, frozen reference policy, minimization loss, negative objective

7.4.3 Why GRPO is cheaper than PPO

This section explains why GRPO is cheaper than PPO when both are compared at the same number of sampled completions.

Key terms: GRPO, PPO, sampled completions, learned value model, value function

7.5 Reinforcement learning with verifiable rewards

Introduces reinforcement learning with verifiable rewards (RLVR), explaining how deterministic checks provide reward signals and where that approach cannot replace other alignment signals.

Key terms: reinforcement learning with verifiable rewards, RLVR, deterministic checker, learned reward model, objectively verifiable, open-ended dialog helpfulness, preference data, human evaluation

7.5.1 Designing a verifier

This section explains how verifier design balances correctness with formatting flexibility and how its reward rules shape model behavior.

Key terms: verifier, gold answer, symbolic equivalence, unit normalization, structured answer extraction, reward function, rule-based rewards, correctness reward, format reward, reasoning/final-answer structure

7.5.2 Avoiding overstatement

This section limits the claim for reinforcement learning with verifiable rewards (RLVR) and explains when to use verifiers versus learned reward models.

Key terms: RLVR, deterministic verifier, hybrid reward design, rule-based rewards, learned reward models, preference judgments

7.6 How reasoning emerges in DeepSeek-R1-Zero

Explains what DeepSeek-R1-Zero’s reinforcement-learning experiment shows about emerging reasoning behavior, and where that conclusion has limits.

Key terms: DeepSeek-R1-Zero, DeepSeek-V3-Base, supervised chain-of-thought examples, verifiable reasoning tasks, aha moment, <think> tag, reasoning region, reasoning-like behavior, verifiable tasks, poor readability

7.7 DeepSeek-R1, R1-Zero, and Distill

Distinguishes the three related DeepSeek-R1 artifacts by how they are produced and what role each serves.

Key terms: DeepSeek-R1-Zero, DeepSeek-R1, DeepSeek-R1-Distill, verifiable rewards, cold-start supervised fine-tuning

7.7.1 From R1-Zero to R1

This subsection explains how DeepSeek-R1 develops from filtered R1-Zero outputs through supervised fine-tuning and reinforcement learning stages, rather than by simply continuing R1-Zero training.

Key terms: R1-Zero, R1, data flow, parameter flow, reasoning traces, cold-start supervised data, chain-of-thought, reasoning-oriented RL, rule-based rewards, language-consistency reward

7.7.2 Why cold-start data matters

Explains why DeepSeek-R1 uses a small supervised cold-start dataset before reasoning RL to establish readable, structured responses and make RL training more effective.

Key terms: cold-start, supervised fine-tuning, reinforcement learning, R1-Zero, cold-start dataset, final answer region, verifier, RL signal, rollouts

7.7.3 Rejection sampling as data construction

Explains how the R1 pipeline turns selected outputs from an RL-trained model into supervised data for a more balanced checkpoint.

Key terms: rejection sampling, candidate responses, supervised training data, RL-trained model, rule-based verifier, data flow, balanced checkpoint

7.7.4 Restoring general assistant behavior

This section explains why verifiable-reward training alone may narrow an assistant’s behavior and how DeepSeek-R1 uses broader supervised data to preserve general usefulness.

Key terms: verifiable reasoning, general assistant, narrow reward, mixed supervised data, reasoning examples, non-reasoning instruction data, pipeline

7.7.5 The final hybrid reward stage

Explains why DeepSeek-R1’s final reinforcement-learning stage combines verifiable and preference-based rewards, and how those rewards complement supervised training.

Key terms: hybrid reward, RLVR, rule-based checker, learned reward model, preference-based signal, verifiable rewards, supervised data, preference-oriented rewards

7.7.6 Distillation into smaller models

This section explains how DeepSeek-R1 transfers reasoning behavior to smaller models through supervised fine-tuning, and why that approach complements rather than replaces RL.

Key terms: DeepSeek-R1-Distill, supervised dataset, reasoning traces, fine-tuned, RL, supervised fine-tuning, distillation, reasoning benchmarks, teacher, student

7.8 Minimal chapter code

This section explains why the chapter uses a small GRPO code example rather than a production RL trainer.

Key terms: verifier rewards, group-relative advantages, clipped GRPO loss, reference-policy KL, training-step skeleton

7.8.1 A minimal verifier

This subsection describes a toy math-answer verifier and explains why it is insufficient for production training.

Key terms: deterministic verifier, gold answer, RLVR

7.8.2 Computing group-relative advantages

Shows how GRPO turns rewards for multiple completions of each prompt into group-relative advantage signals.

Key terms: group-relative baseline, group dimension, normalized advantages, binary verifier rewards

7.8.3 The clipped GRPO loss

This section identifies the inputs to the clipped GRPO loss and explains why the illustrative listing is not a production-ready trainer.

Key terms: current policy, old policy, frozen reference policy, advantages, log probabilities, rollout, attention masks, response masks, policy loss, padding tokens

7.8.4 One training-step skeleton

Shows how one GRPO training step connects sampling, verification, group-relative advantages, policy scoring, loss computation, and an optimizer update.

Key terms: group of completions, verifiable reward, group-relative advantages, GRPO loss, optimizer step, sample_group, sequence_logprobs, old-policy token log probabilities, current policy, frozen reference model

7.9 A worked GRPO example

A four-completion arithmetic example shows how GRPO turns group-relative rewards into a policy update and why final-answer rewards require many training examples.

Key terms: raw rewards, group mean, positive advantages, negative advantages, clipping, KL control, verifier, final answers, RLVR, reasoning patterns

7.9.1 What happens to each token?

Explains how a whole-answer score affects each generated token in a GRPO update, and why the full objective differs from a simplified policy-gradient equation.

Key terms: verifier, completion-level advantage, token log probabilities, credit-assignment signal, policy-gradient equation, GRPO objective, clipping, reference-policy KL, group-relative advantages

7.9.2 Why multiple samples for the same prompt matter

Explains why GRPO compares multiple completions for each prompt and how group size trades off comparison quality against rollout cost.

Key terms: GRPO, group structure, local comparison, completions, global baseline, group-relative baseline, advantage, prompt-aware, group size, prompt-specific reward distribution

7.10 From toy code to a real LLM trainer

Maps the chapter’s minimal GRPO code to a real LLM trainer, identifying the generation, scoring, masking, and infrastructure it leaves out.

Key terms: sample_group, sequence_logprobs, old-policy log probability, PPO/GRPO ratio, padding, masks, loss, algorithmic data flow, distributed execution, rollout generation

7.10.1 Response format and verifier reliability

Explains why reliable RLVR rewards depend on response formats that verifiers can parse consistently.

Key terms: RLVR, verifier, structured final-answer format, format rewards, strict output protocols

7.10.2 trainiMonitoring behavior

This section explains how to monitor GRPO/RLVR training so rising reward or longer responses are not mistaken for better reasoning.

Key terms: average reward, pass rate, response length, KL divergence, invalid-format rate, duplicate-output rate, hidden-test performance, visible-test reward, held-out set, robust answer checking

7.10.3 What changes from PPO to GRPO in code?

Explains which parts of a PPO trainer remain in GRPO, which code paths change, and why the rollout structure must change too.

Key terms: advantage, value model, group normalization, clipped objective, reference KL, multiple completions per prompt, group baseline, rollout structure, KL control

7.11 What does this chapter add to the book?

This section connects the chapter’s post-training hooks to the distinction between using reinforcement learning to strengthen reasoning and using distillation to transfer it.

Key terms: verifier function, group-advantage function, GRPO loss, training-step skeleton, reinforcement learning, distillation, teacher, students, supervised fine-tuning, DeepSeek-R1-Zero

7.12 A practical implementation checklist

This section turns a minimal GRPO script into a checklist for building a reliable reinforcement-learning experiment.

Key terms: task distribution, gold answers, response protocol, verifier, reference policy, KL anchor, prompt group, old-policy log probabilities, response masks, reward

7.13 Reading DeepSeek-R1 claims carefully

Separates evidence-backed DeepSeek-R1 claims from overclaims about verifiable rewards, GRPO, supervision, and distillation.

Key terms: GRPO, verifiable rewards, DeepSeek-R1-Zero, RLVR, value model, learned value baseline, group-relative reward normalization, critic, clipping, R1-Zero

7.14 Common implementation pitfalls

Identifies implementation pitfalls that matter when scaling a compact GRPO training example.

Key terms: response masking, prompt tokens, padding tokens, loss, reward parsing, verifier, RLVR, reward hacking, final answer, KL control

7.15 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 6: The DeepSeek training pipeline: Building a foundation model Chapter 8: Knowledge distillation: Making powerful models practical →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.