A sequence-level importance ratio assigns one importance-sampling weight to an entire generated response. It replaces independently clipped token ratios when the reward is defined at response level, reducing unstable updates from isolated extreme tokens or fragmented clipping. section 6.2.9
For response a_i, GSPO computes the geometric mean of current-to-old policy probabilities across its tokens. Length normalization keeps the clipping range on a comparable per-token average scale across responses of different lengths, while the resulting weight is applied uniformly to every token. section 6.2.9
Equation 6.40
\[\label{eq:response_factorization} \pi_\theta(a \mid s) = \prod_{t=1}^{|a|} \pi_\theta(a_t \mid s, a_{<t}).\]Autoregressive factorization shows how token probabilities combine into the probability of a complete response. section 6.2.9
Equation 6.41
\[\label{eq:GSPO_ratio} \rho_i(\theta) = \left( \frac{\pi_\theta(a_i \mid s)}{\pi_{\theta_{\text{old}}}(a_i \mid s)} \right)^{\frac{1}{|a_i|}} = \exp\left( \frac{1}{|a_i|} \sum_{t=1}^{|a_i|} \log \frac{\pi_\theta(a_{i,t} \mid s, a_{i,<t})}{\pi_{\theta_{\text{old}}}(a_{i,t} \mid s, a_{i,<t})} \right).\]The ratio equation expresses sequence-level weighting as an exponentiated mean of token log-ratios. section 6.2.9
Equation 6.42
\[\label{eq:GSPO_objective} \begin{aligned}[b] J_{\mathrm{GSPO}}(\theta) &= \mathbb{E}_{s \sim \mathcal{D}, \{a_i\}_{i=1}^G \sim \pi_{\theta_{\mathrm{old}}}(\cdot \mid s)} \Biggl[ \frac{1}{G} \sum_{i=1}^G \min\Bigl( \rho_i(\theta) A_i, \\[2.5pt] &\qquad\qquad \operatorname{clip}(\rho_i(\theta), 1-\varepsilon, 1+\varepsilon) A_i \Bigr) \Biggr]. \end{aligned}\]The GSPO objective inserts the sequence-level ratio into the clipped policy objective while retaining group-relative advantages. section 6.2.9
