liveBook · Manning

Softmax

As explained in Understanding AI

Softmax is a function that converts a set of real-valued scores into nonnegative weights whose sum is one.

When a model compares one token with all the others, it needs more than raw scores: it needs weights that show how much attention each token receives. Softmax turns those scores into nonnegative weights that add up to one. section 2.1.3

Turning relevance into attention

For the token "it" in "The performance was unbelievable — the crowd couldn't believe it," dot products produce relevance scores. Softmax converts them into weights such as performance → 0.72, crowd → 0.18, and couldn't → 0.06, with the remaining tokens receiving very little. The model then uses these weights to combine the tokens' value vectors. section 2.1.3

Figure 2.3 When processing "it," the model assigns an attention weight to every other token in the sentence. "Performance" receives the highest weight by far (0.68 out of 1.0)which is how the model infers what "it" refers to.

Visualizes how normalized attention weights distribute focus across every token while processing "it." section 2.1.3

How Understanding AI covers it

How deep it goes
Defines the normalization role and illustrates its output, but does not derive the softmax formula or implementation.
Examples
Uses the sentence about unbelievable performance and assigns example weights to performance, crowd, and couldn't.
What it assumes
Assumes familiarity with attention, token representations, dot products, and query-key comparisons.
What you can do after
Evaluate how attention scores become a probability-like distribution over tokens.
How it connects
Connects softmax to attention, dot products, query vectors, key vectors, and value vectors.

Also explained in

Compare how 26 books explain softmax →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Understanding AI. The section links point to the parts of the book it draws on.