Dynamic sampling policy optimization (DAPO) adapts group relative policy optimization (GRPO) for reasoning-model training. It targets long traces, stronger exploration, and weak learning signals by changing clipping, sampling, loss aggregation, and length handling. section 6.5 section 7.3.3
DAPO combines four operational changes:
- Use separate lower and upper clip limits, ε_low and ε_high, allowing larger positive log-ratio updates that increase exploration. section 6.5
- Apply dynamic sampling by removing batches whose samples all receive reward 0 or all receive reward 1, because they provide no learning signal. section 6.5
- Compute the policy-gradient loss per token rather than treating an entire reasoning trace as one unit. section 6.5
- Add a soft penalty for overly long samples to discourage learning from answers likely truncated by the generation limit. section 6.5
Operational effect
A batch in which every completion earns reward 0 or every completion earns reward 1 is discarded rather than optimized. A useful batch instead supplies relative reward variation, while asymmetric clipping permits larger probability increases for promising tokens and the length penalty suppresses truncated traces. section 6.5
Together, these changes shift GRPO toward exploration and token-level credit assignment while accounting for the unusually long and variable outputs produced by reasoning models. section 6.5 section 7.3.3
