Softmax turns a list of real-valued scores into nonnegative values that add up to 1. These outputs can be read as a distribution of attention or prediction: each value shows a score’s share of the whole. section 11.20
In attention, softmax creates a pie chart of token relevance. Before applying it, scores are scaled because dot products from larger dimensions can become too large and make softmax behave poorly. section 11.20
The calculation follows a short sequence:
- Scale the scores according to their dimensionality. section 11.20
- Exponentiate each scaled score, making every result positive and magnifying differences. section 11.20
- Divide each exponential by the sum of all exponentials, producing values between 0 and 1 whose total is 1. section 11.20
From attention scores to weights
Suppose token scores become 0.1555, 0.0221, and 1.0000 after exponentiation, while the total is 2.46. Softmax assigns weights of about 0.0632, 0.0090, and 0.4064. The largest score receives the largest share, but every token retains a normalized value.
The same idea converts language-model logits—raw scores for vocabulary tokens—into probabilities. A model can then compare possible next tokens, although meaningful predictions require substantial training rather than a single small batch. section 11.29
