Importance sampling estimates an expectation under a target distribution from samples drawn under a different sampling distribution. For a function f, it rewrites Eₚ[f(x)] as E_q[f(x) · p(x)/q(x)]. The ratio p(x)/q(x) is the importance weight: it increases the contribution of samples more likely under the target distribution and decreases those less likely. This makes previously collected data usable after the distribution of interest has changed. section 6.2.9
Equation 6.39
\[\label{eq:IS_identity} \mathbb{E}_{p}[f(x)] = \mathbb{E}_{q}\left[f(x) \frac{p(x)}{q(x)}\right],\]The identity separates the distribution that supplied samples, q, from the distribution whose expectation is needed, p, and corrects their mismatch with a probability ratio. section 6.2.9
In policy-gradient reinforcement learning, the target distribution is the current policy πθ and the sampling distribution is the older policy πθ_old that generated a rollout batch. The policy ratio πθ(a|s)/πθ_old(a|s) reweights old actions when estimating gradients for the current policy. It enables multiple gradient steps per batch, but ratios can drift as optimization changes the policy. section 6.2.9 section 6.2.5
Practical policy optimization modifies this correction to control unstable weights:
- PPO clips the policy ratio near 1 in its surrogate objective, limiting destructive policy updates when current and old policies diverge. section 6.2.5
- Token-level clipping can yield zero gradient for a token outside the clipping interval; rare but important reasoning tokens may then fail to become more probable. section 6.2.9
- GSPO uses one sequence-level ratio, formed from the response probability ratio and length-normalized in log space, rather than separate token ratios. section 6.2.9
- Truncated importance sampling caps a weight with min(ρ, C). Its one-sided cap accepts small bias to bound variance and prevent extreme upweighting. section 6.3.4
Correcting two mismatches in asynchronous RL
A rollout generator and learner can produce different effective token distributions even with identical parameters, because kernels, precision, or parallelism differ. A learner/sampler ratio corrects that implementation-induced gap; it can be multiplied with the usual current-policy/old-policy ratio that corrects policy drift across repeated updates. In a TIS implementation, subtract learner and sampler log-probabilities, clamp the log ratio, exponentiate it, cap at C, and multiply the per-token policy-gradient loss by the detached weight. section 6.3.4
Importance sampling is therefore not a guarantee of stable reuse: large ratios create high-variance updates, while clipping can bias the objective or suppress gradients. The implementation choice is tied to the mismatch being corrected—policy drift within a batch, numerical divergence between sampler and learner, or both—and to whether token-level or sequence-level aggregation is appropriate. section 6.3.4 section 6.6
