liveBook · Manning

Importance weight

As explained in Applied Reinforcement Learning

A probability ratio in PPO is the updated policy's probability for an action divided by the old policy's probability for that action.

In PPO, an importance weight compares how likely the updated policy is to choose an action with how likely the old policy was. For language generation, the state is the prompt plus generated tokens, and the action is the next token. section 10.7

A token becomes more likely

If the old policy assigns “doctor” probability 0.20 and the updated policy assigns 0.24, the ratio is 1.20: that token is 20% more likely in that context. A ratio of 4.0 would indicate a much larger policy change. section 10.7

rt(θ) = πθ(at | st) / πold(at | st).

The ratio compares the updated and old policies’ probabilities for the same action in the same state. section 10.7

How Applied Reinforcement Learning covers it

How deep it goes
Defines the ratio and works through token-probability examples, then motivates clipping without deriving its objective.
Examples
Compares old and updated probabilities of 0.20 and 0.24 for the token “doctor,” then contrasts a ratio of 1.20 with 4.0.
What you can do after
Understand how a probability ratio indicates the size of a policy change for a particular action.
How it connects
Connects the ratio to PPO, policy updates, reinforcement learning from human feedback, and advantage estimates.

Also explained in

Compare how 5 books explain importance weight →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.