Data exploration is the deliberate use of varied actions or strategies to discover which choices produce desirable outcomes. In language-model reinforcement learning, exploration comes from sampling actions rather than always selecting the highest-probability token. section 7.2.1
Exploration operates across generated trajectories:
- A model may sample a lower-probability token instead of taking argmax, preserving opportunities to test alternative actions. section 7.2.1
- Each sampled token forms part of a trajectory: a sequence of states and actions whose outcome receives a reward. section 7.2.1
- Policy-gradient training uses the sampled action’s log probability to increase or decrease its likelihood according to the trajectory’s outcome. section 7.2.1
- Exploration can exceed a small model’s capacity: it may repeat or lose reasoning coherence instead of retaining diverse successful strategies. section 8.7.3
Sampling the next token
Given probabilities of 0.42 for “has,” 0.23 for “owns,” and 0.11 for “will,” the model usually samples “has” but can select either alternative. That variation tests whether a different continuation produces a better trajectory reward. section 7.2.1
Figure 8.27 Performance (AIME 2024) scales smoothly with model size for distilled models. Qwen-Math bases consistently outperform Llama bases at similar sizes, showing that the base model’s pre-existing capabilities interact with distillation quality.

Use the performance figure to compare how distilled models scale with size and how base-model capabilities interact with distillation quality. section 8.7.3














