Softmax turns a vector of scores into nonnegative weights that sum to 1. For scores z, each weight is exp(zᵢ) / Σⱼ exp(zⱼ): exponentiate each score, sum the results, and divide each result by that sum. section 8.3.2 section 4.2.4
The resulting weights serve several purposes:
- Attention: Apply softmax row by row to scaled, causally masked scores. Multiply the weights by the value matrix to produce context vectors. section 2.2.1
- Token prediction: Convert the final logits vector into vocabulary probabilities, then select the highest-probability token with argmax. section 2.2.3
- Mixture-of-experts routing: Keep the top-k expert scores for each token, mask the rest, and normalize the selected scores to weight expert outputs. section 4.2.4
Mask excluded scores with negative infinity before softmax. Setting them to zero is not enough: exp(0) is 1, so those positions would still receive positive weight. Negative infinity produces zero weight, preventing attention to future tokens or selection of discarded experts. section 2.2.1 section 4.2.4
Temperature changes the distribution
For class logits [5, 3, 1, −1], standard softmax gives approximately [0.865, 0.117, 0.016, 0.002]. Dividing the logits by temperature T=3 before softmax produces [0.523, 0.268, 0.138, 0.071]. This softer distribution makes relationships among the less likely classes more visible for knowledge distillation; at T=10, the distribution becomes nearly uniform and the signal is diluted. section 8.3.2
Figure 8.8 The same logits [5.0, 3.0, 1.0, -1.0] processed through softmax at three temperatures. At T=1, the distribution is too peaked, at T=3 the dark knowledge emerges clearly, and at T=10 the signal is lost in near-uniformity.

Compare the same logits at three temperatures to see the distribution flatten. section 8.3.2

























