Distribution mismatch occurs when training data come from one probability distribution while optimization targets another. In policy-gradient training, rollouts generated by the old policy differ from the current policy after updates. Importance sampling reweights samples from the sampling policy to estimate expectations under the target policy. section 6.2.9
In reinforcement learning, mismatch appears when:
- Multiple gradient steps move the current policy away from the policy that generated a fixed rollout batch. section 6.2.9
- Asynchronous generation and training systems hold stale or different model weights. section 6.2.9 section 6.6
- Offline distillation trains on teacher-generated sequences even though the student will generate different sequences at inference time. section 12.3.2
Policy correction
Let p be the target distribution and q the sampling distribution. Importance sampling estimates an expectation under p by weighting samples from q with p(x)/q(x). PPO and GRPO apply this correction per token and clip the surrogate objective; GSPO instead uses a sequence-level ratio, while CISPO clips importance weights so tokens retain gradient signals. section 6.2.9
Clipping stabilizes optimization but can give zero gradient to tokens whose ratios leave the clipping range, including rare reasoning steps. On-policy distillation addresses a related mismatch by sampling from the student and comparing its next-token distribution with the teacher, producing a reverse-KL objective on student-induced states. section 6.2.9 section 12.3.2
