A log probability is the natural logarithm of the probability a policy assigns to an action. In policy gradients, its gradient—the score function—points toward parameter changes that make that action more likely. section 9.2
Log probabilities help policy learning in two ways:
- They turn products of probabilities into sums and keep tiny probabilities manageable. section 9.2
- Their gradients indicate how to make sampled actions more or less likely, guided by the rewards they produce. section 9.3
A Gaussian action
For a Gaussian policy with mean 0.7 and standard deviation 0.2, the sampled action 0.82 has log probability 0.51; action 0.3 has log probability −1.31. The latter is less likely under the policy. section 9.3
Listing 10.8 Computing Log Probabilities
def _compute_log_probs(
self,
logits: torch.Tensor,
input_ids: torch.Tensor,
) -> torch.Tensor:
shift_logits = logits[:, :-1, :].contiguous() #A
shift_labels = input_ids[:, 1:].contiguous() #B
log_probs = F.log_softmax(shift_logits, dim=-1) #C
token_log_probs = log_probs.gather(
dim=-1,
index=shift_labels.unsqueeze(-1),
).squeeze(-1) #D
return token_log_probs.mean(dim=-1) #ESee how generated-token log probabilities are computed from model logits. section 10.8.5









