Importance weight
An importance weight is a probability ratio that reweights an outcome or sample to account for a difference between the distribution that generated it and a target distribution.
Reinforcement Learning from Human Feedback
A probability ratio is the quotient of the probability assigned to an outcome by a target policy or distribution and the probability assigned by a sampling or reference policy or distribution.
- How it is used
- Corrects policy-distribution mismatch when reusing rollout batches or training asynchronously.
- How deep it goes
- Derives the sampling identity and policy ratios, then compares token- and sequence-level calculations and clipping strategies.
- Examples
- Uses per-token log-probability differences and clipped losses in a PPO implementation.
- What you can do after
- Evaluate how ratio granularity and clipping placement affect gradient signals and stability.
Read how Reinforcement Learning from Human Feedback explains importance weight →
Applied Reinforcement Learning
A probability ratio in PPO is the updated policy's probability for an action divided by the old policy's probability for that action.
- How it is used
- Uses the probability ratio in PPO to track policy changes while improving responses in reinforcement learning from human feedback.
- How deep it goes
- Defines the ratio and works through token-probability examples, then motivates clipping without deriving its objective.
- Examples
- Compares old and updated probabilities of 0.20 and 0.24 for the token “doctor,” then contrasts a ratio of 1.20 with 4.0.
- What you can do after
- Understand how a probability ratio indicates the size of a policy change for a particular action.
Read how Applied Reinforcement Learning explains importance weight →
AI Model Evaluation
An importance weight is a factor that reweights an observation to account for a difference between the distribution that generated it and a target distribution.
- How it is used
- Uses importance weights to estimate a new policy’s performance from outcomes logged under an old policy.
- How deep it goes
- Defines the policy probability ratio and demonstrates weighted evaluation, with practical checks for unstable estimates.
- Examples
- Uses a 10%-to-30% recommendation probability change and a movie evaluation with click and watch-time rewards.
- What you can do after
- Evaluate whether weighted offline policy estimates are sufficiently stable for a decision.
Machine Learning Algorithms in Depth
An importance weight is a numerical factor that adjusts the contribution of a sample or outcome in a weighted estimate.
- How it is used
- Adjusts contributions from proposal samples in Monte Carlo integration and from sampled self-avoiding walks in weighted distance estimates.
- How deep it goes
- Defines the target-to-proposal ratio, gives a sample-and-weight procedure, and shows a separate stepwise walk weight.
- Examples
- Uses a sine-function expectation with a Chi-shaped target and Gaussian proposal, plus weighted squared distances from lattice walks.
- What you can do after
- Build it from scratch by sampling a proposal, calculating target-to-proposal ratios, and averaging weighted function values.
Read how Machine Learning Algorithms in Depth explains importance weight →
Optimization Algorithms: AI techniques for design, planning, and control problems
A probability ratio is the quotient of an action probability under an updated policy and its probability under an old policy.
- How it is used
- Controls policy-gradient updates in PPO by comparing action probabilities before and after a policy change.
- How deep it goes
- Defines ratio direction and clipping behavior, but stops short of deriving the full surrogate objective.
- Examples
- Uses sampled state-action probabilities, ratios above or below one, and an example clipping parameter of 0.2.
- What you can do after
- Evaluate PPO policy changes by interpreting action-probability ratios and their clipping range.
Also covered in
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
What to know first
- policy gradient methodA policy gradient method is a reinforcement-learning method that optimizes a parameterized policy by estimating the gradient of expected return with respect to its parameters.
- advantage functionAn advantage function measures the expected return of an action relative to a baseline, typically the expected return under a policy at the same state.
- Group Relative Policy OptimizationGroup Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled actions or outputs.
- policyA policy is a rule, principle, or decision function that governs behavior or determines actions or permissions under specified conditions.
Related concepts
- importance samplingproduced by · Importance sampling is a statistical technique that estimates expectations under a target distribution by reweighting samples drawn from another distribution.
- Proximal Policy Optimizationused by · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
- CISPOused by · CISPO is a policy-optimization algorithm that clips importance weights directly while preserving gradient signals for all tokens.
- clipped surrogate objectiveused by · A clipped surrogate objective is an optimization objective that clips a policy probability ratio to limit its contribution and constrain policy updates.
- inverse propensity weightingused by · Inverse propensity weighting estimates causal effects or counterfactual outcomes by weighting observed outcomes inversely to the probability of the observed treatment or action given covariates or context.
Questions about importance weight
What is importance weight?
An importance weight is a probability ratio that reweights an outcome or sample to account for a difference between the distribution that generated it and a target distribution.
Which Manning books explain importance weight?
Reinforcement Learning from Human Feedback, Applied Reinforcement Learning, AI Model Evaluation, Machine Learning Algorithms in Depth and Optimization Algorithms: AI techniques for design, planning, and control problems.
What should I know before learning importance weight?
Manning books that teach importance weight usually introduce policy gradient method, advantage function, Group Relative Policy Optimization and policy first.
What is importance weight also called?
Importance weight is also called policy ratio.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.




