Log probability is the natural logarithm of a model-assigned token probability. Applying log scaling preserves likelihood ordering while replacing tiny products with more manageable sums: higher probabilities become values closer to zero, and unlikely tokens become more negative. section 5.5 section 5.4
For next-token scoring, logits are converted with torch.log_softmax, then indexed at the observed next tokens. Summing those selected values gives a sequence log probability, equivalent to taking the logarithm of the product of conditional token probabilities. section 5.5 section 6.8
Choose the aggregation that matches the scoring or training objective:
- Average only the answer-token log probabilities to obtain a length-normalized score for comparing responses of different lengths. section 5.6
- Sum the answer-token log probabilities to represent the likelihood of an entire generated sequence. section 6.8
- Multiply sequence log probabilities by detached advantages and negate the mean when constructing the GRPO policy-gradient loss for a minimizing optimizer. section 6.9
Scoring Berlin versus Bridge
Given the prompt “What is the capital of Germany?”, score “The capital of Germany is Berlin.” and “The capital of Germany is Bridge.” by excluding prompt tokens and averaging answer-token log probabilities. The model assigns the nonsensical ending a much more negative score, illustrating confidence-based ranking without guaranteeing factual correctness. section 5.6 section 5.4
Figure 5.13 Overview of moving from token probabilities to token log probabilities, which provide a numerically more stable basis for the log-probability scoring used later in self-refinement.
Trace the conversion from token probabilities to numerically stable token log probabilities before using them in a scorer. section 5.5
