liveBook · Manning

Softmax

As explained in Timeless Algorithms, The Foundational Ideas

The softmax function transforms a vector of real-valued scores into a probability distribution.

Softmax converts scores across mutually exclusive, collectively exhaustive outcomes into probabilities that sum to one. This lets models express confidence and rank or sample possible outcomes. In attention, it turns dot products into weights that help construct contextual representations. section 2.4.2 section 12.2

From scores to predictions

A language model applies softmax to scores for vocabulary words to produce probabilities for the next word. In a transformer attention calculation, softmax turns dot products into weights; a worked example uses these steps for pronoun resolution. section 2.4.2 section 12.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains score-to-probability conversion and attention weighting through applications, without detailing the function’s mathematical form.
Examples
Uses class probabilities, next-word prediction, and attention examples involving pronoun resolution, semantic association, and factual recall.
What you can do after
Understand how scores become probabilities for classification and next-token selection, and how softmax weights contribute to attention.
How it connects
Links softmax to logistic regression, random forests, language models, self-attention, and dot products.

Also explained in

Compare how 26 books explain softmax →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.