Softmax converts a set of scores into values between 0 and 1 that sum to 1. Those values can represent probabilities across classes or weights across positions. Unlike sigmoid, which is used here for binary classification, softmax is the preferred output activation for multicategory classification. section 2.4.2 section 9.1.1
Ten clothing categories
A PyTorch classifier ends with `nn.Linear(64, 10)`, producing one logit for each clothing category. Pass those raw logits and labels from 0 to 9 to `nn.CrossEntropyLoss()`; do not add softmax to the model first. The loss combines log-softmax with negative log-likelihood loss. After softmax, the ten values can be read as category probabilities that add up to 1. To choose a predicted category, the testing code instead takes `argmax` of the logits. section 2.4.2
In self-attention, softmax has a different job: it turns scaled query–key scores into attention weights. For “How are you?”, four tokens produce a 4 × 4 weight matrix. Each row distributes one token’s attention across all four tokens, including itself, and totals 100%. Multiplying those weights by the value vectors produces the attention output. section 9.1.1
Figure 9.2 Steps to calculate attention weights. The input embedding is passed through two neural networks to obtain query Q and key K. The scaled attention scores are calculated as the dot product of Q and K divided by the square root of the dimension of K. Finally, we apply the softmax function on the scaled attention scores to obtain attention weights, which demonstrate how each element is related to all other elements in the sequence.
Follow the diagram from query–key scores through scaling and softmax to see where attention weights are formed. section 9.1.1
The same scores-to-distribution step appears in text generation: a GPT-2XL linear head produces logits for 50,257 vocabulary tokens, and softmax can turn them into a distribution over the next token. section 11.2.4
