liveBook · Manning

Softmax

As explained in Build a DeepSeek Model (From Scratch)

Softmax is a function that converts a vector of scores into nonnegative values summing to one.

Softmax turns a vector of scores into nonnegative weights that sum to 1. For scores z, each weight is exp(zᵢ) / Σⱼ exp(zⱼ): exponentiate each score, sum the results, and divide each result by that sum. section 8.3.2 section 4.2.4

The resulting weights serve several purposes:

  • Attention: Apply softmax row by row to scaled, causally masked scores. Multiply the weights by the value matrix to produce context vectors. section 2.2.1
  • Token prediction: Convert the final logits vector into vocabulary probabilities, then select the highest-probability token with argmax. section 2.2.3
  • Mixture-of-experts routing: Keep the top-k expert scores for each token, mask the rest, and normalize the selected scores to weight expert outputs. section 4.2.4

Mask excluded scores with negative infinity before softmax. Setting them to zero is not enough: exp(0) is 1, so those positions would still receive positive weight. Negative infinity produces zero weight, preventing attention to future tokens or selection of discarded experts. section 2.2.1 section 4.2.4

Temperature changes the distribution

For class logits [5, 3, 1, −1], standard softmax gives approximately [0.865, 0.117, 0.016, 0.002]. Dividing the logits by temperature T=3 before softmax produces [0.523, 0.268, 0.138, 0.071]. This softer distribution makes relationships among the less likely classes more visible for knowledge distillation; at T=10, the distribution becomes nearly uniform and the signal is diluted. section 8.3.2

Figure 8.8 The same logits [5.0, 3.0, 1.0, -1.0] processed through softmax at three temperatures. At T=1, the distribution is too peaked, at T=3 the dark knowledge emerges clearly, and at T=10 the signal is lost in near-uniformity.

Compare the same logits at three temperatures to see the distribution flatten. section 8.3.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Works through exponentiation and normalization numerically, and explains masking and temperature scaling in practical settings.
Examples
Uses four-class image logits at three temperatures and top-k expert scores to demonstrate normalization.
What you can do after
Evaluate how masking and temperature affect normalized weights in attention, expert routing, and distillation.
How it connects
Links softmax to causal attention, top-k expert routing, argmax token selection, temperature scaling, and knowledge distillation.

Also explained in

Compare how 26 books explain softmax →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.