Importance weight

As explained in Optimization Algorithms: AI techniques for design, planning, and control problems

A probability ratio is the quotient of an action probability under an updated policy and its probability under an old policy.

An importance weight compares how likely the same action is under an updated policy and an older policy: r(θ) = updated probability ÷ old probability. A value above 1 means the action became more likely; a value between 0 and 1 means it became less likely. section 12.1.5

In PPO-CLIP, the ratio scales the policy-gradient objective while clipping limits the effective update to 1 − ε through 1 + ε. With ε = 0.2, the intended range is 0.8 to 1.2, helping prevent an excessively large policy change. section 12.1.5

Reading a ratio

For a sampled state and action, a ratio of 1.15 means the updated policy assigns 15% more probability to that action. A ratio of 0.7 means it assigns less probability than before; PPO may clip its contribution when the ratio moves beyond the allowed range. section 12.1.5

Figure 12.6 Clipped surrogate objective in PPO

Use the clipped-surrogate-objective figure to see how probability ratios and advantage estimates determine PPO’s update behavior. section 12.1.5

How Optimization Algorithms: AI techniques for design, planning, and control problems covers it

How deep it goes

Defines ratio direction and clipping behavior, but stops short of deriving the full surrogate objective.

Examples

Uses sampled state-action probabilities, ratios above or below one, and an example clipping parameter of 0.2.

What it assumes

Assumes probability distributions and policy-gradient terminology while defining the ratio’s directional meaning.

What you can do after

Evaluate PPO policy changes by interpreting action-probability ratios and their clipping range.

How it connects

Connects probability ratios with PPO, policy gradients, advantage estimates, clipping, and KL divergence.

liveBookmanning.com

Read Optimization Algorithms: AI techniques for design, planning, and control problems on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackA probability ratio is the quotient of the probability assigned to an outcome by a target policy or distribution and the probability assigned by a sampling or reference policy or distribution.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthAn importance weight is a numerical factor that adjusts the contribution of a sample or outcome in a weighted estimate.

Compare how 3 books explain importance weight →

Related concepts

• importance samplingproduced by · Importance sampling is a statistical technique that estimates expectations under a target distribution by reweighting samples drawn from another distribution.
• Proximal Policy Optimizationused by · Proximal Policy Optimization is a policy-gradient reinforcement-learning algorithm that limits policy updates to keep successive policies close and promote stable learning.
• CISPOused by · CISPO is a policy-optimization algorithm that clips importance weights directly while preserving gradient signals for all tokens.
• inverse propensity weightingused by · Inverse propensity weighting estimates causal effects or counterfactual outcomes by weighting observed outcomes inversely to the probability of the observed treatment or action given covariates or context.

This explainer was generated by Manning's knowledge graph from the text of Optimization Algorithms: AI techniques for design, planning, and control problems. The section links point to the parts of the book it draws on.