A probability distribution converts temperature-scaled next-token logits into normalized scores between 0 and 1 whose total is 1. These scores represent relative confidence and provide the sampling basis for choosing the next token. section 4.4.3
Build the distribution during inference:
- Scale the next-token logits by the selected temperature. section 4.4.3
- Apply torch.softmax along the vocabulary dimension. section 4.4.3
- Verify that the resulting probabilities sum to 1, then sample with torch.multinomial. section 4.4.3
Temperature changes the distribution
At temperature 5.0, the token " Berlin" has probability 0.0003. Lowering temperature to 0.5 raises it to 0.3398 and concentrates probability on fewer tokens; repeated sampling demonstrates the resulting frequency differences. section 4.4.3
Figure 4.10 Overview of the sampling process for generating tokens. Here, we focus on steps 3.3 and 3.4, where the next-token scores are converted into a probability distribution. The next token will be sampled based on that distribution.
Follow the inference workflow from rescaled logits through probability conversion to next-token sampling. section 4.4.3
