liveBook · Manning

Log probability

As explained in Applied Reinforcement Learning

The logarithm of the probability or probability density assigned to an outcome by a probability model.

A log probability is the natural logarithm of the probability a policy assigns to an action. In policy gradients, its gradient—the score function—points toward parameter changes that make that action more likely. section 9.2

Log probabilities help policy learning in two ways:

  • They turn products of probabilities into sums and keep tiny probabilities manageable. section 9.2
  • Their gradients indicate how to make sampled actions more or less likely, guided by the rewards they produce. section 9.3

A Gaussian action

For a Gaussian policy with mean 0.7 and standard deviation 0.2, the sampled action 0.82 has log probability 0.51; action 0.3 has log probability −1.31. The latter is less likely under the policy. section 9.3

Listing 10.8 Computing Log Probabilities
def _compute_log_probs(
    self,
    logits: torch.Tensor,
    input_ids: torch.Tensor,
) -> torch.Tensor:
    shift_logits = logits[:, :-1, :].contiguous()  #A
    shift_labels = input_ids[:, 1:].contiguous()  #B
 
    log_probs = F.log_softmax(shift_logits, dim=-1)  #C
 
    token_log_probs = log_probs.gather(
        dim=-1,
        index=shift_labels.unsqueeze(-1),
    ).squeeze(-1)  #D
 
    return token_log_probs.mean(dim=-1)  #E

See how generated-token log probabilities are computed from model logits. section 10.8.5

How Applied Reinforcement Learning covers it

How deep it goes
Defines log probability and its score-function role, explains numerical benefits, and works through a Gaussian example.
Examples
Uses a 30% action probability and Gaussian actions sampled at 0.82 and 0.3 to illustrate log-probability values.
What you can do after
Evaluate how likely sampled actions or generated tokens are under a policy, and compute token log probabilities from model logits.
How it connects
Connects log probability to policy gradients, score functions, REINFORCE, PPO, and Gaussian policies.

Also explained in

Compare how 10 books explain log probability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.