In PPO, an importance weight compares how likely the updated policy is to choose an action with how likely the old policy was. For language generation, the state is the prompt plus generated tokens, and the action is the next token. section 10.7
A token becomes more likely
If the old policy assigns “doctor” probability 0.20 and the updated policy assigns 0.24, the ratio is 1.20: that token is 20% more likely in that context. A ratio of 4.0 would indicate a much larger policy change. section 10.7
The ratio compares the updated and old policies’ probabilities for the same action in the same state. section 10.7




