Softmax
Softmax is a function that exponentiates a vector of scores and normalizes the results into nonnegative values that sum to one.
Optimization Algorithms: AI techniques for design, planning, and control problems
Softmax is a function that converts a vector of real-valued scores into nonnegative class probabilities that sum to one.
- How it is used
- Converts classifier logits and attention scores into probabilities or normalized coefficients for prediction, loss calculation, and selection.
- How deep it goes
- Shows explicit NumPy normalization in a forward pass and describes attention uses, but does not derive the function mathematically.
- Examples
- Uses penguin-species classification, graph-neighbor attention, and point selection for a convex hull.
- What you can do after
- Apply it via NumPy to convert class logits into probabilities for a neural-network loss calculation.
Server-Side WebAssembly
Softmax is a mathematical transformation that converts a vector of scores into nonnegative probabilities summing to one.
- How it is used
- Interprets ML inference outputs as class probabilities or next-token sampling weights.
- How deep it goes
- Works through exponentiation, summation, normalization, ranking, and weighted sampling without discussing mathematical derivation.
- Examples
- Uses a SqueezeNet Persian-cat prediction and a “Once upon a time” story-generation loop.
- What you can do after
- Apply it via wasi-nn inference outputs to rank image labels or sample generated tokens.
Deep Learning with PyTorch, Second Edition
A normalization function that converts a vector of scores into a probability distribution by exponentiating and renormalizing the values.
- How it is used
- Converts classifier scores into class probabilities and query–key scores into normalized attention weights.
- How deep it goes
- Works through exponentiation, normalization, output values, and the dimension argument without developing deeper mathematical theory.
- Examples
- Uses three-score calculations, batched PyTorch tensors, image classification, and a query–key–value attention implementation.
- What you can do after
- Apply it via PyTorch nn.Softmax or F.softmax to normalize scores along the intended dimension.
Read how Deep Learning with PyTorch, Second Edition explains softmax →
Build a Reasoning Model (From Scratch)
A normalization function that converts logits into a probability distribution over possible next tokens.
- How it is used
- Converts temperature-scaled next-token logits into probabilities used by the sampling step.
- How deep it goes
- Explains normalization, temperature effects, plotting, and PyTorch usage, then stops before deeper mathematical derivation.
- Examples
- Uses the " Berlin" token, probability-sum checks, plots, and repeated multinomial sampling.
- What you can do after
- Apply it via PyTorch to normalize logits and feed probabilities into token sampling.
Read how Build a Reasoning Model (From Scratch) explains softmax →
Math and Architectures of Deep Learning
A smooth function that converts an unnormalized score vector into a probability-like distribution over classes.
- How it is used
- Converts classifier score vectors into class probabilities for prediction and cross-entropy-based training.
- How deep it goes
- Defines score normalization, probability properties, confidence interpretation, and smooth argmax approximation, then connects it to PyTorch training behavior.
- Examples
- Uses cat-dog-airplane-automobile scores, a three-class height-and-weight classifier, MNIST LeNet, and Faster R-CNN regions.
- What you can do after
- Apply it via PyTorch to convert prediction scores into probabilities and select a class with argmax.
Read how Math and Architectures of Deep Learning explains softmax →
Grokking Machine Learning
A function that converts multiple real-valued scores into positive values whose normalized values sum to one and represent class probabilities.
- How it is used
- Produces probability distributions when a classifier must choose among more than two classes.
- How deep it goes
- Explains the motivation, exponential normalization, worked scores, and Keras implementation without deriving deeper optimization details.
- Examples
- Uses dog, cat, and bird scores, a two-class circular dataset, and handwritten MNIST digit classification.
- What you can do after
- Apply softmax in Keras to create multi-class outputs and select predictions from their highest probabilities.
Learn Generative AI with PyTorch
An activation that converts raw scores into a probability distribution over multiple classes or positions.
- How it is used
- Turns multicategory logits into class probabilities, scaled attention scores into weights, and vocabulary logits into token probabilities.
- How deep it goes
- Explains where softmax belongs in worked classification and attention pipelines, without deriving its formula.
- Examples
- Uses a ten-category clothing classifier, a four-token attention matrix, and GPT-2XL vocabulary logits.
- What you can do after
- Apply it via PyTorch cross-entropy without duplicating softmax, and interpret normalized attention weights or token probabilities.
Read how Learn Generative AI with PyTorch explains softmax →
Transformers in Action
The softmax function converts a vector of scores into nonnegative values that sum to one.
- How it is used
- Converts attention scores into weights for combining values and token logits into probabilities used during sampling.
- How deep it goes
- Works through softmax’s role in attention and temperature sampling with code, but does not implement the softmax function itself.
- Examples
- Shows row-wise softmax in a NumPy attention calculation and `temperature=0.7` in a Transformers generation call.
- What you can do after
- Apply it via Transformers `model.generate()` to adjust sampling temperature, and interpret softmax weights in an attention calculation.
Build a Text-to-Image Generator (from Scratch)
A function that converts a vector of real-valued scores into nonnegative values that sum to one.
- How it is used
- Converts scaled query-key scores into attention weights that determine how tokens combine value vectors.
- How deep it goes
- Explains scoring, scaling, normalization, and value aggregation, while deferring detailed multi-head implementation internals.
- Examples
- Uses the sentence “Wie geht es dir,” a four-token embedding, and a PyTorch attention-head implementation.
- What you can do after
- Apply it via scaled dot-product attention to normalize scores and aggregate value vectors.
Read how Build a Text-to-Image Generator (from Scratch) explains softmax →
Natural Language Processing in Action, Second Edition
Softmax is a normalized exponential function that converts a vector of scores into nonnegative values summing to one.
- How it is used
- Uses neural-network output layers to represent probabilities for mutually exclusive classification targets.
- How deep it goes
- Explains the normalized-exponential calculation, rank preservation, probability interpretation, and activation choice for multiclass versus multi-label tasks.
- Examples
- Uses three-dimensional score vectors and skip-gram predictions such as Monet targeting Claude or painted.
- What you can do after
- Evaluate/understand when a classification output should form one shared probability distribution.
Read how Natural Language Processing in Action, Second Edition explains softmax →
Inside Deep Learning: Math, Algorithms, Models
A function that turns a vector of scores into normalized weights over items or classes by exponentiating and renormalizing them.
- How it is used
- Connects neural-network class scores to probability distributions and cross-entropy training.
- How deep it goes
- Explains score conversion, normalization, confidence, terminology, and PyTorch application without deriving deeper statistical justifications.
- Examples
- Uses true-class confidence values, two-class circles-and-crosses data, PyTorch code, and a contour visualization.
- What you can do after
- Apply it via PyTorch to convert model outputs into class probabilities and visualize learned confidence regions.
Read how Inside Deep Learning: Math, Algorithms, Models explains softmax →
Deep Learning with R, Third Edition
A softmax operation converts a vector of scores into nonnegative values that sum to one, typically representing class probabilities.
- How it is used
- Turns neural-network scores into class probabilities for classification and token probabilities for text generation.
- How deep it goes
- Explains probability normalization, demonstrates prediction output, and contrasts explicit softmax with logits without deriving the operation mathematically.
- Examples
- Uses a ten-class MNIST digit classifier and a transformer whose token outputs can be normalized from logits.
- What you can do after
- Apply it via Keras to produce class probabilities or configure cross-entropy to consume logits instead.
Read how Deep Learning with R, Third Edition explains softmax →
Hugging Face in Action
Softmax is a mathematical function that converts a vector of numbers into a probability distribution by exponentiating and normalizing its values.
- How it is used
- Converts model output logits into class or next-token probabilities for classification and transformer decoding.
- How deep it goes
- Defines exponentiation and normalization, notes that probabilities sum to one, and demonstrates application without deriving implementation internals.
- Examples
- Uses neural-network output values, next-token prediction, and CLIP labels for cat, dog, tiger, and train.
- What you can do after
- Apply it via PyTorch's softmax() to rank candidate labels after a CLIP model produces logits.
Machine Learning Algorithms in Depth
A function that converts a vector of scores into nonnegative values summing to one, typically interpreted as class probabilities.
- How it is used
- Converts multiclass classification scores into probabilities and constrains mixture-density-network weights to sum to one.
- How deep it goes
- Defines softmax through multiclass posteriors, but develops the accompanying from-scratch logistic-regression optimization for the binary case.
- Examples
- Uses MNIST digit classification and a synthetic-data Gaussian mixture density network; poses a temperature-parameter exercise.
- What you can do after
- Apply it via Keras/TensorFlow as a CNN classifier output or as constrained mixture weights.
Read how Machine Learning Algorithms in Depth explains softmax →
Data Without Labels
A softmax function converts multiple real-valued outputs into nonnegative values that sum to one, typically representing class probabilities.
- How it is used
- Uses softmax in neural-network output layers to classify inputs among multiple classes.
- How deep it goes
- Defines probability output and prediction selection, but does not detail the mathematical formula or implementation procedure.
- Examples
- Uses car-bike-truck and cat-dog image classification examples to show competing class probabilities.
- What you can do after
- Evaluate multiclass neural-network outputs by selecting the class with the highest softmax probability.
Sutskever's List
A function that converts a vector of scores into nonnegative values summing to one, interpreted as probabilities.
- How it is used
- Converts final-layer class scores into probabilities for choosing a classifier’s predicted category.
- How deep it goes
- Defines the score-to-probability transformation and shows its role in classification without detailing its computation.
- Examples
- Uses AlexNet’s 1,000 ImageNet category scores as the concrete classification example.
- What you can do after
- Understand how raw category scores become predicted probabilities and determine a classification.
Grokking AI Algorithms, Second Edition
The softmax function converts a vector of real-valued scores into nonnegative values that sum to one.
- How it is used
- Converts attention scores and language-model logits into normalized weights or token probabilities.
- How deep it goes
- Works through scaling, exponentiation, normalization, and numeric tables, but does not develop deeper theory.
- Examples
- Uses token attention tables and a 35-token vocabulary probability table to demonstrate normalization and prediction.
- What you can do after
- Evaluate attention relevance and token probabilities by following the softmax calculation from scores to normalized values.
Read how Grokking AI Algorithms, Second Edition explains softmax →
Related concepts
- probability distributionproduces · A probability distribution is a mathematical specification assigning probabilities to the possible values or outcomes of a random variable, with total probability one.
- temperatureused by · A temperature parameter adjusts the concentration of a probability distribution, affecting sampling randomness or the likelihood of accepting less favorable transitions.
- reduction operationuses · A reduction operation combines multiple input values or partial results according to a combining rule to produce fewer results, often a single aggregate.
- probability theoryproduces · Probability theory is the branch of mathematics that formalizes uncertainty and analyzes random events and outcomes using probability measures.
- attentionused by · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
- attention weightproduces · An attention weight is a computed coefficient, often normalized, that determines how strongly an input element or value contributes to an attention-based representation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.