liveBook · Manning

Reinforcement learning

As explained in Build a DeepSeek Model (From Scratch)

Reinforcement learning is a machine-learning paradigm in which an agent improves its behavior by optimizing rewards received from interactions or evaluations.

Reinforcement learning (RL) trains an agent through interaction: it observes a state, takes an action, receives a reward, and moves to a new state. Unlike supervised learning, it need not provide the correct action at every step. A reward can instead arrive after several actions, allowing the agent to discover behaviors that improve the outcome. section 7.1

For language-model reasoning, this solves a practical problem: a final answer may be easy to check even when the desired intermediate tokens are unknown. The model samples complete responses, receives scores, and updates its policy to make rewarded behavior more likely. section 7.0 section 7.1

A language-model rollout maps the RL framework to token generation:

  • State: the prompt plus all tokens generated so far. section 7.1.2
  • Action and transition: sample a next token and append it to the context. section 7.1.2
  • Reward: score the completed response; sampling permits exploration of different responses to the same prompt. section 7.1.2

Rewarding a checkable answer

A prompt asks for the total after adding two cans of three tennis balls to five balls. The correct answer is 11. A verifier can reward a correct final answer without specifying the intermediate reasoning tokens; sampled rollouts let the model try different solution paths. section 7.1.2 section 7.1

Figure 7.3 Mapping reinforcement learning concepts to LLM generation. The growing context is the state; each sampled next token is an action, and the final response receives a reward.

Follow the growing context as the state, each sampled token as an action, and the completed response as the reward target. section 7.1.2

Group Relative Policy Optimization (GRPO) simplifies the training stack by replacing a learned value baseline with group-relative reward normalization. It avoids training a critic while retaining clipping and reference-policy control. With verifiable rewards, GRPO can induce longer, more self-corrective reasoning in a capable base model. section 7.13 section 7.0

Verifiable rewards have limits: deterministic checkers do not resolve open-ended helpfulness, subtle factuality, or safety preferences. Supervised fine-tuning complements RL by teaching known formats and styles, while RL optimizes scored outcomes. section 7.13 section 7.7.2

From R1-Zero to R1

R1-Zero tests pure RL with verifiable rewards, without supervised chain-of-thought data. R1 instead begins with cold-start supervised examples, applies reasoning-oriented RL, filters sampled outputs for supervised fine-tuning, and finishes with RL using rule-based and learned rewards. The cold start addresses readability and language-mixing problems observed in R1-Zero. section 7.7 section 7.7.1 section 7.7.2

Direct RL versus distillation

On the same Qwen2.5-32B base, an R1-distilled model scored 72.6% on AIME 2024 versus 47.0% for a direct RL run. This comparison favors distillation in that experiment, not necessarily in every setting. section 8.7.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Moves from the interaction framework through policy gradients, PPO and GRPO to PyTorch training excerpts, while excluding massive-scale distributed infrastructure.
Examples
Uses a tennis-ball arithmetic rollout, the R1-Zero and R1 pipelines, and a Qwen2.5-32B comparison with distillation.
What it assumes
Starts with minimal prior-knowledge assumptions and maps basic reinforcement learning concepts onto a pretrained language model.
What you can do after
Build a PyTorch GRPO training path using verifier rewards and training steps.
How it connects
Connects reinforcement learning with supervised fine-tuning, policy gradients, GRPO, verifiable rewards and knowledge distillation.

Also explained in

Compare how 28 books explain reinforcement learning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.