liveBook · Manning

Data exploration

As explained in Build a DeepSeek Model (From Scratch)

Exploration is the process of trying varied actions or strategies to discover which ones produce desirable outcomes during learning.

Data exploration is the deliberate use of varied actions or strategies to discover which choices produce desirable outcomes. In language-model reinforcement learning, exploration comes from sampling actions rather than always selecting the highest-probability token. section 7.2.1

Exploration operates across generated trajectories:

  • A model may sample a lower-probability token instead of taking argmax, preserving opportunities to test alternative actions. section 7.2.1
  • Each sampled token forms part of a trajectory: a sequence of states and actions whose outcome receives a reward. section 7.2.1
  • Policy-gradient training uses the sampled action’s log probability to increase or decrease its likelihood according to the trajectory’s outcome. section 7.2.1
  • Exploration can exceed a small model’s capacity: it may repeat or lose reasoning coherence instead of retaining diverse successful strategies. section 8.7.3

Sampling the next token

Given probabilities of 0.42 for “has,” 0.23 for “owns,” and 0.11 for “will,” the model usually samples “has” but can select either alternative. That variation tests whether a different continuation produces a better trajectory reward. section 7.2.1

Figure 8.27 Performance (AIME 2024) scales smoothly with model size for distilled models. Qwen-Math bases consistently outperform Llama bases at similar sizes, showing that the base model’s pre-existing capabilities interact with distillation quality.

Use the performance figure to compare how distilled models scale with size and how base-model capabilities interact with distillation quality. section 8.7.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains stochastic sampling, trajectory-level rewards, and capacity limits, then contrasts exploration with imitation through distillation.
Examples
Uses alternative next-token probabilities, sampled response trajectories, repetitive small-model reasoning, and distilled reasoning traces.
What you can do after
Evaluate when stochastic sampling enables discovery and when distillation replaces exploration for capacity-constrained models.
How it connects
Connects exploration with reinforcement learning, policy gradients, trajectories, distillation, imitation, and reasoning strategies.

Also explained in

Compare how 15 books explain data exploration →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.