liveBook · Manning

Softmax

As explained in Math and Architectures of Deep Learning

A smooth function that converts an unnormalized score vector into a probability-like distribution over classes.

Softmax turns a classifier’s unbounded per-class scores into a probability vector: every value lies between 0 and 1, the values sum to 1, and each position estimates the probability of its class. The largest probability identifies the predicted class, while the full vector expresses relative confidence. section 9.1.5 section 9.4

Unlike a hard argmax-one-hot choice, softmax changes smoothly as scores change. Two nearly identical score vectors can fall on opposite sides of an argmax decision boundary and yield completely different one-hot outputs; their softmax outputs remain close. This differentiability makes the probability conversion suitable for training. section 9.1.5

Reading class confidence

For cat, dog, airplane, and automobile scores such as [9.99, 10, 0.01, -10], softmax assigns nearly equal probability to cat and dog, with dog slightly higher, and near-zero probability to the other classes. The predicted class is dog, but the near tie signals low confidence between dog and cat. section 9.1.5

In multiclass training, softmax commonly feeds cross-entropy loss. A prediction assigning a dog probability of 0.502 for a dog image has high loss, whereas one assigning 0.997 has low loss. PyTorch combines these operations in cross-entropy loss for better numerical behavior, so training receives raw scores rather than separately computed softmax probabilities. section 9.1.6 section 9.2.3

Figure 9.2 Two-class argmaxonehot and softmax (function of score vector [s0, s1]) on the (s0, s1) plane. The decision boundary is the 45o line s0 = s1.

(a) Step function: z = 1 if s0 > = s1, else z = 0

(b) Softmax: differential approximation of the step function

(c) Step function: z = 1 if s1 > = s0, else z = 0

(d) Softmax: differential approximation of the step function

Compare the abrupt two-class argmax decision boundary with softmax’s smooth probability surfaces. section 9.1.5

At prediction time, a model can apply softmax to its scores and then take argmax. This pattern appears in a three-class height-and-weight classifier, LeNet’s 10-class digit output, and Faster R-CNN’s class estimates for proposed image regions. section 9.2.3 section 11.1 section 11.4

How Math and Architectures of Deep Learning covers it

How deep it goes
Defines score normalization, probability properties, confidence interpretation, and smooth argmax approximation, then connects it to PyTorch training behavior.
Examples
Uses cat-dog-airplane-automobile scores, a three-class height-and-weight classifier, MNIST LeNet, and Faster R-CNN regions.
What it assumes
Assumes class score vectors and labels, then defines their conversion into probability vectors.
What you can do after
Apply it via PyTorch to convert prediction scores into probabilities and select a class with argmax.
How it connects
Connects softmax to logits, argmax, cross-entropy loss, decision boundaries, and neural-network classifiers.
Cover of Math and Architectures of Deep Learning
liveBookmanning.com
Read Math and Architectures of Deep Learning on liveBook Get the book at manning.com

Also explained in

Compare how 17 books explain softmax →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Math and Architectures of Deep Learning. The section links point to the parts of the book it draws on.