liveBook · Manning

DeepSeek-R1-Zero

As explained in Build a DeepSeek Model (From Scratch)

DeepSeek-R1-Zero is a reasoning-focused language model trained to develop reasoning behavior through reinforcement learning without conventional supervised fine-tuning as its initial reasoning-training stage.

DeepSeek-R1-Zero tests whether a capable base model can develop reasoning behavior through reinforcement learning without supervised chain-of-thought data. Starting from DeepSeek-V3-Base, it receives rewards for solving verifiable reasoning tasks rather than imitating human-written solution traces. section 7.6

The training pressure changes response behavior: difficult problems receive longer attempts, allowing the model to try approaches, inspect intermediate results, detect errors, and revise. The reported “aha moment” illustrates explicit strategy correction, while longer responses matter only when they support better solutions. section 7.6 section 7.13

Interpret the result with these boundaries:

  • Treat <think> as a formatting convention separating reasoning-style text from the final answer, not as a Python construct or interpreter feature. section 7.6
  • Expect rough edges: R1-Zero can produce poor readability and language mixing, motivating later refinement. section 7.6
  • Distinguish R1-Zero from DeepSeek-R1: R1-Zero is the pure RL experiment, whereas R1 adds supervised stages, filtering, mixed instruction data, and final preference optimization. section 7.7 section 7.7.1
  • Read the transition from R1-Zero to R1 as data flow: filtered outputs become training data rather than transferring R1-Zero parameters directly. section 7.7.1

A self-correction trace

A reasoning response begins with one approach, notices that the path is flawed, and changes strategy before producing the answer. This behavior exemplifies the reported “aha moment”; the visible reasoning region is marked with <think>. section 7.6

Figure 7.14 DeepSeek-R1 training pipeline, adapted from Guo et al., DeepSeek-R1, Figure 2. The flow from R1-Zero into R1 represents filtered training data, not direct parameter transfer.

Use the pipeline diagram to trace how R1-Zero outputs are sampled, filtered, and converted into training data for the more usable R1 system. section 7.7.1

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Traces reward-driven behavior and training consequences, but stops short of specifying the complete optimization implementation.
Examples
Uses the aha-moment trace, <think> formatting, and accuracy-versus-response-length training curves as concrete illustrations.
What it assumes
Assumes familiarity with language models and reinforcement learning while grounding verifiable rewards and reasoning-format conventions.
What you can do after
Evaluate claims about emergent reasoning while distinguishing verifiable-task gains from general alignment or polished assistant behavior.
How it connects
Connects reinforcement learning, GRPO, RLVR, verifiable rewards, supervised fine-tuning, reward models, and knowledge distillation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.