Sequence log probability scores a complete response by summing the model’s log probabilities for its answer tokens, excluding prompt tokens. Less-negative scores indicate responses the model considers more likely; the score supports rollout evaluation in GRPO and connects directly to token-level training losses. section 6.8 section 8.6
Compute the score through the causal language-model output sequence:
- Run the model on the concatenated prompt and response token IDs, then apply log_softmax to obtain token distributions. section 6.8
- Select each response token’s probability from the preceding position, beginning at prompt_len - 1. section 6.8
- Sum the selected log probabilities; torch.gather provides a compact GPU-friendly implementation equivalent to explicit indexing. section 6.8
- Average the sum by the number of answer tokens when comparing examples of different lengths, as required for cross-entropy-style losses. section 8.6
Comparing four rollouts
The responses “83”, “The correct answer is 83”, “The final answer is 83”, and “We get 38” receive scores of -7.9243, -20.1546, -16.6130, and -23.3677. The concise correct answer scores highest, while the incorrect answer scores lowest. section 6.8
Figure 8.12 Illustration of token and sequence log probabilities. The log probabilities of the correct next tokens are summed to obtain the sequence log probability, which is the basis for the cross-entropy loss used later in this chapter.
Trace how correct next-token log probabilities are summed into a sequence score and then used as the basis for cross-entropy. section 8.6
