liveBook · Manning

Softmax

As explained in Grokking Machine Learning, Second Edition

The softmax function converts a vector of real-valued scores into positive values that sum to one, yielding a probability distribution over classes.

Softmax converts several classifier scores into positive values that add up to one, so the result can be read as a probability distribution across classes. It extends the sigmoid idea from two classes to multiple classes. section 6.13 section 10.15

Start with raw scores rather than separate yes-or-no decisions. Apply the exponential function to every score, then divide each resulting value by their total. Exponentials keep every value positive while preserving the ordering of the scores. section 6.13

Dog, cat, or bird?

Suppose three classifiers produce scores of 3 for dog, 2 for cat, and −1 for bird. Exponentiating gives approximately 20.085, 7.389, and 0.368. Normalizing these values produces roughly 72% dog, 27% cat, and 1% bird. The largest score remains the most likely class, but the output also expresses uncertainty. section 6.13

Softmax is useful when:

  • Several classes compete for one prediction, such as dog, cat, and bird. section 6.13
  • A neural network needs one probability for each output class. section 10.15
  • Classifiers need to be combined into one distribution instead of returning only a discrete label. section 6.14

In a Keras neural network, softmax can serve as the final activation for an output layer with one node per class. Categorical labels and categorical cross-entropy then support training the classifier. section 10.18

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Explains exponential transformation, normalization, worked values, and neural-network placement without deriving training internals.
Examples
Uses dog, cat, and bird scores, plus a Keras network classifying a two-feature dataset with two softmax outputs.
What it assumes
Assumes familiarity with perceptrons and sigmoid-based logistic classifiers while introducing multiclass scoring from the normalization problem.
What you can do after
Apply it via Keras as a multiclass output activation and interpret normalized class probabilities.
How it connects
Connects softmax to sigmoid, logistic classifiers, linear classifiers, neural networks, and categorical cross-entropy.

Also explained in

Compare how 26 books explain softmax →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.