GSPO addresses a failure mode in PPO and GRPO: token-level importance ratios can become unstable, clip many tokens independently, or let one extreme token dominate a response update. This fragments learning when rewards are assigned to complete sequences, especially for long outputs and sparse mixture-of-experts models. section 6.2.9
GSPO extends GRPO by applying importance sampling at completion level. For response aᵢ, it computes one length-normalized ratio as the geometric mean of token-policy ratios: ρᵢ(θ) = exp((1/|aᵢ|) Σₜ log[πθ(aᵢ,ₜ|s,aᵢ,<ₜ) / πold(aᵢ,ₜ|s,aᵢ,<ₜ)]). The length normalization prevents response length from biasing the ratio, while sequence-level aggregation matches sequence-level rewards. section 6.2.9
For G completions sampled from the old policy, GSPO maximizes the average clipped surrogate: Σᵢ min(ρᵢAᵢ, clip(ρᵢ, 1−ε, 1+ε)Aᵢ) / G. Aᵢ is the group-relative advantage used by GRPO. Clipping therefore constrains each completion’s update rather than separately suppressing tokens within it. section 6.2.9
A reasoning completion with a rare key token
Suppose a response contains a rare but important reasoning step. Token-level clipping can drop that token’s gradient or fragment the response signal when other tokens clip differently. GSPO instead assigns the response a single geometric-mean importance weight, preserving a coherent completion-level update. section 6.2.9
Equation 6.41
\[\label{eq:GSPO_ratio} \rho_i(\theta) = \left( \frac{\pi_\theta(a_i \mid s)}{\pi_{\theta_{\text{old}}}(a_i \mid s)} \right)^{\frac{1}{|a_i|}} = \exp\left( \frac{1}{|a_i|} \sum_{t=1}^{|a_i|} \log \frac{\pi_\theta(a_{i,t} \mid s, a_{i,<t})}{\pi_{\theta_{\text{old}}}(a_{i,t} \mid s, a_{i,<t})} \right).\]Inspect the formal length-normalized geometric-mean ratio used to aggregate token probabilities into one completion-level importance weight. section 6.2.9
Equation 6.42
\[\label{eq:GSPO_objective} \begin{aligned}[b] J_{\mathrm{GSPO}}(\theta) &= \mathbb{E}_{s \sim \mathcal{D}, \{a_i\}_{i=1}^G \sim \pi_{\theta_{\mathrm{old}}}(\cdot \mid s)} \Biggl[ \frac{1}{G} \sum_{i=1}^G \min\Bigl( \rho_i(\theta) A_i, \\[2.5pt] &\qquad\qquad \operatorname{clip}(\rho_i(\theta), 1-\varepsilon, 1+\varepsilon) A_i \Bigr) \Biggr]. \end{aligned}\]Inspect the clipped GSPO surrogate objective and its averaging across grouped completions. section 6.2.9
