liveBook · Manning

Importance weight

Also called policy ratio

An importance weight is a probability ratio that reweights an outcome or sample to account for a difference between the distribution that generated it and a target distribution.

How 5 Manning books explain it

Reinforcement Learning from Human Feedback

A probability ratio is the quotient of the probability assigned to an outcome by a target policy or distribution and the probability assigned by a sampling or reference policy or distribution.

How it is used
Corrects policy-distribution mismatch when reusing rollout batches or training asynchronously.
How deep it goes
Derives the sampling identity and policy ratios, then compares token- and sequence-level calculations and clipping strategies.
Examples
Uses per-token log-probability differences and clipped losses in a PPO implementation.
What you can do after
Evaluate how ratio granularity and clipping placement affect gradient signals and stability.

Read how Reinforcement Learning from Human Feedback explains importance weight →

Applied Reinforcement Learning

A probability ratio in PPO is the updated policy's probability for an action divided by the old policy's probability for that action.

How it is used
Uses the probability ratio in PPO to track policy changes while improving responses in reinforcement learning from human feedback.
How deep it goes
Defines the ratio and works through token-probability examples, then motivates clipping without deriving its objective.
Examples
Compares old and updated probabilities of 0.20 and 0.24 for the token “doctor,” then contrasts a ratio of 1.20 with 4.0.
What you can do after
Understand how a probability ratio indicates the size of a policy change for a particular action.

Read how Applied Reinforcement Learning explains importance weight →

AI Model Evaluation

An importance weight is a factor that reweights an observation to account for a difference between the distribution that generated it and a target distribution.

How it is used
Uses importance weights to estimate a new policy’s performance from outcomes logged under an old policy.
How deep it goes
Defines the policy probability ratio and demonstrates weighted evaluation, with practical checks for unstable estimates.
Examples
Uses a 10%-to-30% recommendation probability change and a movie evaluation with click and watch-time rewards.
What you can do after
Evaluate whether weighted offline policy estimates are sufficiently stable for a decision.

Read how AI Model Evaluation explains importance weight →

Machine Learning Algorithms in Depth

An importance weight is a numerical factor that adjusts the contribution of a sample or outcome in a weighted estimate.

How it is used
Adjusts contributions from proposal samples in Monte Carlo integration and from sampled self-avoiding walks in weighted distance estimates.
How deep it goes
Defines the target-to-proposal ratio, gives a sample-and-weight procedure, and shows a separate stepwise walk weight.
Examples
Uses a sine-function expectation with a Chi-shaped target and Gaussian proposal, plus weighted squared distances from lattice walks.
What you can do after
Build it from scratch by sampling a proposal, calculating target-to-proposal ratios, and averaging weighted function values.

Read how Machine Learning Algorithms in Depth explains importance weight →

Optimization Algorithms: AI techniques for design, planning, and control problems

A probability ratio is the quotient of an action probability under an updated policy and its probability under an old policy.

How it is used
Controls policy-gradient updates in PPO by comparing action probabilities before and after a policy change.
How deep it goes
Defines ratio direction and clipping behavior, but stops short of deriving the full surrogate objective.
Examples
Uses sampled state-action probabilities, ratios above or below one, and an example clipping parameter of 0.2.
What you can do after
Evaluate PPO policy changes by interpreting action-probability ratios and their clipping range.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains importance weight →

Also covered in

What to know first

Related concepts

Questions about importance weight

What is importance weight?

An importance weight is a probability ratio that reweights an outcome or sample to account for a difference between the distribution that generated it and a target distribution.

Which Manning books explain importance weight?

Reinforcement Learning from Human Feedback, Applied Reinforcement Learning, AI Model Evaluation, Machine Learning Algorithms in Depth and Optimization Algorithms: AI techniques for design, planning, and control problems.

What should I know before learning importance weight?

Manning books that teach importance weight usually introduce policy gradient method, advantage function, Group Relative Policy Optimization and policy first.

What is importance weight also called?

Importance weight is also called policy ratio.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.