Importance weight
An importance weight is the ratio of an outcome’s probability under a target distribution or policy to its probability under a sampling or reference distribution or policy, used to reweight that outcome.
Reinforcement Learning from Human Feedback
A probability ratio is the quotient of the probability assigned to an outcome by a target policy or distribution and the probability assigned by a sampling or reference policy or distribution.
- How it is used
- Corrects policy-gradient updates when current-policy optimization reuses rollouts sampled from an older policy.
- How deep it goes
- Derives the expectation-reweighting identity and traces token- and sequence-level clipping choices, including their gradient consequences.
- Examples
- Uses PPO’s per-token log-probability implementation and GSPO’s length-normalized sequence ratio as concrete formulations.
- What you can do after
- Apply it via PPO-style losses to reweight old-policy rollouts while bounding unstable policy updates.
Read how Reinforcement Learning from Human Feedback explains importance weight →
Machine Learning Algorithms in Depth
An importance weight is a numerical factor that adjusts the contribution of a sample or outcome in a weighted estimate.
- How it is used
- Adjusts contributions from proposal samples in Monte Carlo integration and from sampled self-avoiding walks in weighted distance estimates.
- How deep it goes
- Defines the target-to-proposal ratio, gives a sample-and-weight procedure, and shows a separate stepwise walk weight.
- Examples
- Uses a sine-function expectation with a Chi-shaped target and Gaussian proposal, plus weighted squared distances from lattice walks.
- What you can do after
- Build it from scratch by sampling a proposal, calculating target-to-proposal ratios, and averaging weighted function values.
Read how Machine Learning Algorithms in Depth explains importance weight →
Optimization Algorithms: AI techniques for design, planning, and control problems
A probability ratio is the quotient of an action probability under an updated policy and its probability under an old policy.
- How it is used
- Controls policy-gradient updates in PPO by comparing action probabilities before and after a policy change.
- How deep it goes
- Defines ratio direction and clipping behavior, but stops short of deriving the full surrogate objective.
- Examples
- Uses sampled state-action probabilities, ratios above or below one, and an example clipping parameter of 0.2.
- What you can do after
- Evaluate PPO policy changes by interpreting action-probability ratios and their clipping range.
Related concepts
- importance samplingproduced by · Importance sampling is a statistical technique that estimates expectations under a target distribution by reweighting samples drawn from another distribution.
- Proximal Policy Optimizationused by · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
- CISPOused by · CISPO is a policy-optimization algorithm that clips importance weights directly while preserving gradient signals for all tokens.
- inverse propensity weightingused by · Inverse propensity weighting estimates causal effects or counterfactual outcomes by weighting observed outcomes inversely to the probability of the observed treatment or action given covariates or context.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.