CISPO stabilizes reinforcement-learning updates when token-level importance-sampling ratios become noisy, especially in large-scale mixture-of-experts models. Instead of clipping the surrogate objective as PPO and GRPO do, it clips the importance weights themselves and applies a stop gradient to those weights. Every token therefore retains a gradient proportional to its advantage, while clipping limits how strongly the importance ratio amplifies or suppresses that signal. section 6.2.10 section 6.6
For rollout tokens sampled from the previous policy, let ρᵢ,ₜ(θ) be the new-to-old policy probability ratio and Aᵢ,ₜ its advantage. CISPO clips the ratio to ρ̂ᵢ,ₜ between 1 − ε_low and 1 + ε_high, then optimizes the token log-probability multiplied by stop-gradient(ρ̂ᵢ,ₜ)Aᵢ,ₜ. The stop gradient uses the clipped weight without differentiating through it. section 6.2.10
Equation 6.43
\[\label{eq:CISPO_objective} \begin{aligned}[b] J_{\mathrm{CISPO}}(\theta) &= \mathbb{E}_{s \sim \mathcal{D}, \{a_i\}_{i=1}^K \sim \pi_{\theta_{\mathrm{old}}}(\cdot \mid s)} \Biggl[ \frac{1}{\sum_{i=1}^K |a_i|} \sum_{i=1}^K \sum_{t=1}^{|a_i|} \operatorname{sg}\!\left(\hat{\rho}_{i,t}(\theta)\right) A_{i,t} \\[2pt] &\qquad\qquad {} \log \pi_\theta(a_{i,t} \mid s, a_{i,<t}) \Biggr], \end{aligned}\]The objective averages token-level log-probability updates across sampled completions, weighting each token by its stopped clipped ratio and advantage. section 6.2.10
Equation 6.44
\[\label{eq:CISPO_ratio} \hat{\rho}_{i,t}(\theta) = \text{clip}\left( \rho_{i,t}(\theta), 1 - \varepsilon_{\text{low}}, 1 + \varepsilon_{\text{high}} \right), \quad \rho_{i,t}(\theta) = \frac{\pi_\theta(a_{i,t} \mid s, a_{i,<t})}{\pi_{\theta_{\text{old}}}(a_{i,t} \mid s, a_{i,<t})}.\]The ratio compares current and rollout-policy token probabilities and supports unequal lower and upper clipping bounds. section 6.2.10
CISPO changes the optimization trade-off in three practical ways:
- Preserves gradient signals for tokens whose ratios would otherwise be clipped away by a surrogate-objective method. section 6.2.10
- Introduces bias by clipping weights, but controls gradient variance caused by unstable per-token ratios and off-policy updates. section 6.2.10 section 6.6
- Allows asymmetric bounds; a higher upper limit can permit larger updates for tokens the model should upweight and may encourage exploration. section 6.2.10
CISPO versus PPO and GRPO
PPO and GRPO clip a token-level surrogate objective with bilateral bounds, potentially eliminating gradient contributions outside the clipping range. CISPO instead clips the token importance weight, stops its gradient, and continues updating the token through its advantage. GSPO changes the granularity to a sequence-level ratio, while CISPO remains token-level. section 6.2.11 section 6.2.10
