liveBook · Manning

Log probability

As explained in Reinforcement Learning from Human Feedback

A log probability is the logarithm of the probability assigned to an event or token by a probabilistic model.

A log probability is the logarithm of a probability assigned to an event or token. In policy-gradient implementations, differences between new and reference log probabilities provide a convenient route to per-token probability ratios. section 6.3.1

Advantages determine how the policy uses log probabilities during optimization:

  • Positive advantage: increase the log ratio, making the sampled completion more likely. section 6.3.1
  • Negative advantage: reduce the likelihood of the sampled completion when the updated policy made it more likely. section 6.3.1
  • Zero advantage: produce zero loss and leave the policy unchanged. section 6.3.1

Policy-gradient loss

A compact implementation uses pg_loss = -advantages * ratio. The ratio is commonly computed from the difference between new and reference log probabilities; summed token log probabilities indicate how likely the completion is under the model. section 6.3.1

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Explains token-level log-probability differences, summed completion likelihoods, and their effects under three advantage cases.
Examples
Uses positive, negative, and zero advantages alongside the pg_loss = -advantages * ratio implementation.
What you can do after
Apply it via a policy-gradient loss to reinforce, suppress, or preserve sampled completions.
How it connects
Links probability ratios with policy gradients, advantages, PPO, and GRPO.
Cover of Reinforcement Learning from Human Feedback
liveBookmanning.com
Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

Compare how 8 books explain log probability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.