An importance weight reuses samples drawn from one distribution to estimate an expectation under another. For target distribution p and sampling distribution q, the weight for x is p(x)/q(x): samples become more influential when the target assigns them higher probability, and less influential when it assigns them lower probability. section 6.2.9
Equation 6.39
\[\label{eq:IS_identity} \mathbb{E}_{p}[f(x)] = \mathbb{E}_{q}\left[f(x) \frac{p(x)}{q(x)}\right],\]The identity multiplies each sample’s contribution by the target-to-sampling probability ratio, correcting distribution mismatch without resampling. section 6.2.9
In policy gradients, the target is the current policy πθ and the sampling policy is πθold, which generated the rollout. The policy ratio ρ(θ)=πθ(a|s)/πθold(a|s) therefore corrects gradients after the current policy has drifted from the policy that collected the data. This enables multiple gradient steps on one batch and also compensates for stale weights between asynchronous inference and training systems. section 6.2.9 section 6.2.5
The ratio has direct optimization meaning:
- A ratio of 1 means the action is equally likely under current and sampling policies; ratios above or below 1 respectively upweight or downweight its contribution. section 6.2.5
- With positive advantage, increasing the log-ratio makes a beneficial action more likely; with negative advantage, the loss pushes a harmful action to become less likely. section 6.3.1
- For language models, ratios may be computed per token from a difference in new and old log-probabilities. section 6.3.1 section 6.3.5
PPO: bound a reused-rollout update
PPO multiplies an advantage by the token-level ratio, then clips the ratio to [1−ε, 1+ε] in its surrogate objective. The bound limits policy drift and prevents destructive updates when the new and old probabilities diverge. Outside that range, the clipped PPO objective can become flat for a token, producing zero gradient in the affected direction. section 6.2.5 section 6.2.6
rewards = (
rewards - self.beta * per_token_kl
) # Shape: (B*G, L) 1
values = value_net(completions) # Shape: (B*G, L)
returns = torch.zeros_like(rewards)
running = torch.zeros(
rewards.shape[0],
device=rewards.device,
dtype=rewards.dtype,
)
for t in reversed(range(rewards.shape[1])):2
running = (
rewards[:, t] + self.gamma * running
) * completion_mask[:, t]
returns[:, t] = running
advantages = returns - values.detach() # Shape: (B*G, L)3
advantages = (
advantages - advantages.mean()
) / (advantages.std() + 1e-8)
ratio = torch.exp(
new_per_token_logps - per_token_logps
) # Shape: (B*G, L) 4
eps = self.cliprange # e.g. 0.2
pg_losses1 = -advantages * ratio # Shape: (B*G, L)
pg_losses2 = -advantages * torch.clamp(
ratio,
1.0 - eps,
1.0 + eps,
) # Shape: (B*G, L)
pg_loss_max = torch.max(pg_losses1, pg_losses2) # Shape: (B*G, L)
vf_loss = 0.5 * ((returns - values) ** 2) # Shape: (B*G, L)5
per_token_loss = (
pg_loss_max + self.vf_coef * vf_loss
) # Shape: (B*G, L) 6
loss = (
(per_token_loss * completion_mask).sum(dim=1)
/ completion_mask.sum(dim=1)
).mean()
with torch.no_grad():7
clip_frac = (
(pg_losses2 > pg_losses1).float() * completion_mask
).sum() / completion_mask.sum()
approx_kl = (
0.5
* ((new_per_token_logps - per_token_logps) ** 2)
* completion_mask
).sum() / completion_mask.sum()
value_loss = vf_loss.mean()
The implementation exponentiates a log-probability difference to obtain the ratio, then compares unclipped and clipped advantage-weighted losses. section 6.3.5
Clipping location changes the trade-off. PPO and GRPO clip the surrogate objective at token level; GSPO instead forms a length-normalized sequence-level ratio. CISPO clips the importance weight itself and stops gradients through that weight, retaining an advantage-proportional gradient for every token while accepting bias to control variance. These choices matter when per-token ratios are numerically unstable or when clipping would drop rare but important tokens. section 6.2.9 section 6.2.10 section 6.2.11
