Numerical stability keeps probability scoring accurate when values become extremely small. Here, token probabilities are converted to log probabilities with logarithmic scaling to avoid underflow and rounding issues during sequence scoring. section 5.5
The treatment builds the concept through these steps:
- Starts from token probabilities used to rank responses, where joint probabilities can become tiny. section 5.5
- Applies torch.log or torch.log_softmax to move into log space while preserving ordering. section 5.5
Token scoring with log probabilities
A simple logit-to-probability example computes softmax probabilities, then converts them with torch.log and torch.log_softmax. The plotted values keep the same ranking, and a next-token example scores "Berlin" higher than "Bridge".
