Softmax
Softmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.
CUDA for LLMs
Softmax is a function that converts real-valued scores into nonnegative values that sum to one.
- How it is used
- Uses softmax to normalize classifier logits and transformer attention scores while motivating efficient GPU reductions.
- How deep it goes
- Develops stable formulas and naive CUDA kernels into online, shared-memory, and tiled attention implementations.
- Examples
- Works through four logits numerically and implements row-wise CUDA kernels with running normalization statistics.
- What you can do after
- Builds softmax from scratch using stable CPU and CUDA implementations, then applies online and shared-memory optimizations.
Optimization Algorithms: AI techniques for design, planning, and control problems
Softmax is a function that converts a vector of real-valued scores into nonnegative class probabilities that sum to one.
- How it is used
- Converts classifier outputs to probabilities and normalizes attention scores for graph and pointer networks.
- How deep it goes
- Shows exponentiation and row-wise normalization in NumPy, then uses the resulting probabilities in a loss calculation.
- Examples
- Uses penguin species classification and convex hull point selection to illustrate probability outputs.
- What you can do after
- Apply it in NumPy to obtain class probabilities and calculate negative log-likelihood.
Server-Side WebAssembly
Softmax is a mathematical transformation that converts a vector of scores into nonnegative probabilities summing to one.
- How it is used
- Interprets ML inference outputs as class probabilities or next-token sampling weights.
- How deep it goes
- Works through exponentiation, summation, normalization, ranking, and weighted sampling without discussing mathematical derivation.
- Examples
- Uses a SqueezeNet Persian-cat prediction and a “Once upon a time” story-generation loop.
- What you can do after
- Apply it via wasi-nn inference outputs to rank image labels or sample generated tokens.
Deep Learning with PyTorch, Second Edition
A normalization function that converts a vector of scores into a probability distribution by exponentiating and renormalizing the values.
- How it is used
- Converts classifier scores into probabilities and attention scores into normalized weights.
- How deep it goes
- Explains exponentiation and normalization with worked vectors, then demonstrates dimension selection and model integration.
- Examples
- Uses score vectors, a bird-versus-airplane classifier, and a dot product attention implementation.
- What you can do after
- Apply it via PyTorch's nn.Softmax or F.softmax to normalize scores along a chosen dimension.
Read how Deep Learning with PyTorch, Second Edition explains softmax →
Build a Reasoning Model (From Scratch)
A normalization function that converts logits into a probability distribution over possible next tokens.
- How it is used
- Converts temperature-scaled next-token logits into probabilities used by the sampling step.
- How deep it goes
- Explains normalization, temperature effects, plotting, and PyTorch usage, then stops before deeper mathematical derivation.
- Examples
- Uses the " Berlin" token, probability-sum checks, plots, and repeated multinomial sampling.
- What you can do after
- Apply it via PyTorch to normalize logits and feed probabilities into token sampling.
Read how Build a Reasoning Model (From Scratch) explains softmax →
Math and Architectures of Deep Learning
A smooth function that converts an unnormalized score vector into a probability-like distribution over classes.
- How it is used
- Converts classifier score vectors into class probabilities for prediction and cross-entropy-based training.
- How deep it goes
- Defines score normalization, probability properties, confidence interpretation, and smooth argmax approximation, then connects it to PyTorch training behavior.
- Examples
- Uses cat-dog-airplane-automobile scores, a three-class height-and-weight classifier, MNIST LeNet, and Faster R-CNN regions.
- What you can do after
- Apply it via PyTorch to convert prediction scores into probabilities and select a class with argmax.
Read how Math and Architectures of Deep Learning explains softmax →
Rearchitecting LLMs
Softmax converts a vector of scores into nonnegative probabilities that sum to one.
- How it is used
- Converts attention scores into value weights and teacher logits into soft labels for distillation.
- How deep it goes
- Explains normalization and a PyTorch distillation workflow but does not derive the softmax formula.
- Examples
- Uses logits [2.0, 1.0, 0.1] and an attention query for “spring” to illustrate normalization.
- What you can do after
- Apply it via PyTorch alongside log_softmax and KL divergence to compute a distillation logits loss.
Grokking Machine Learning
A function that converts multiple real-valued scores into positive values whose normalized values sum to one and represent class probabilities.
- How it is used
- Produces probability distributions when a classifier must choose among more than two classes.
- How deep it goes
- Explains the motivation, exponential normalization, worked scores, and Keras implementation without deriving deeper optimization details.
- Examples
- Uses dog, cat, and bird scores, a two-class circular dataset, and handwritten MNIST digit classification.
- What you can do after
- Apply softmax in Keras to create multi-class outputs and select predictions from their highest probabilities.
Learn Generative AI with PyTorch
An activation that converts raw scores into a probability distribution over multiple classes or positions.
- How it is used
- Converts classifier logits, attention scores, and language-model logits into normalized probabilities or weights.
- How deep it goes
- Explains normalization through classifier and attention workflows, without deriving the softmax formula.
- Examples
- Uses a ten-class clothing classifier, a four-token attention example, and GPT-2XL vocabulary logits.
- What you can do after
- Apply it via PyTorch loss functions when training a multicategory classifier and interpret normalized attention weights.
Read how Learn Generative AI with PyTorch explains softmax →
Applied Reinforcement Learning
Softmax converts a vector of real-valued scores into nonnegative probabilities that sum to one.
- How it is used
- Maps policy logits to discrete-action probabilities and attention scores to weights.
- How deep it goes
- Defines the score normalization and works through a numerical action example and attention’s saturation concern.
- Examples
- Uses factory production logits and temperature-zone query-key scores as examples.
- What you can do after
- Apply softmax to logits or similarity scores to obtain action probabilities or normalized attention weights.
Grokking Machine Learning, Second Edition
The softmax function converts a vector of real-valued scores into positive values that sum to one, yielding a probability distribution over classes.
- How it is used
- Turns competing classifier scores into a probability distribution for multiclass prediction.
- How deep it goes
- Explains exponential transformation, normalization, worked values, and neural-network placement without deriving training internals.
- Examples
- Uses dog, cat, and bird scores, plus a Keras network classifying a two-feature dataset with two softmax outputs.
- What you can do after
- Apply it via Keras as a multiclass output activation and interpret normalized class probabilities.
Read how Grokking Machine Learning, Second Edition explains softmax →
Build a DeepSeek Model (From Scratch)
Softmax is a function that converts a vector of scores into nonnegative values summing to one.
- How it is used
- Converts scores into weights for attention, token prediction, expert routing, and knowledge distillation.
- How deep it goes
- Works through exponentiation and normalization numerically, and explains masking and temperature scaling in practical settings.
- Examples
- Uses four-class image logits at three temperatures and top-k expert scores to demonstrate normalization.
- What you can do after
- Evaluate how masking and temperature affect normalized weights in attention, expert routing, and distillation.
Read how Build a DeepSeek Model (From Scratch) explains softmax →
Build AI Drug Discovery Pipelines
Softmax is a function that converts a vector of scores into nonnegative values summing to one.
- How it is used
- Deploys probabilities for multiclass CYP inhibition and next-token selection during SMILES molecule generation.
- How deep it goes
- Explains class-specific weights, logits, probability conversion, prediction, and cross-entropy, while flagging the single-class assumption.
- Examples
- Uses five CYP inhibition classes, SMILES vocabulary prediction, and temperature-based molecule generation as concrete applications.
- What you can do after
- Apply it via multiclass models and molecular decoders to interpret class probabilities or sample the next SMILES character.
Read how Build AI Drug Discovery Pipelines explains softmax →
GPU Programming with Triton
Softmax normalization exponentiates scores and divides by their sum to produce nonnegative values that sum to one.
- How it is used
- Uses softmax as a chained-reduction case study for classification logits and attention similarities.
- How deep it goes
- Explains maximum-shift stability and outlines an unfused Triton pipeline with global reductions and intermediate materialization.
- Examples
- Works through logits [1000, 1001, 1002] and contrasts FP16 with FP32 range and precision.
- What you can do after
- Evaluate numerical stability and memory costs by tracing maximum, exponentiation, sum, and normalization.
Transformers in Action
The softmax function converts a vector of scores into nonnegative values that sum to one.
- How it is used
- Uses softmax to produce token-sampling probabilities and normalize attention scores.
- How deep it goes
- Explains temperature effects and the attention-weighted sum, with library calls rather than a derivation of softmax.
- Examples
- Demonstrates generation at temperature 0.7 and attention over input embeddings using query, key, and value matrices.
- What you can do after
- Apply it via Transformers generation settings to adjust sampling randomness.
Build a Text-to-Image Generator (from Scratch)
A function that converts a vector of real-valued scores into nonnegative values that sum to one.
- How it is used
- Converts scaled query-key scores into attention weights that determine how tokens combine value vectors.
- How deep it goes
- Explains scoring, scaling, normalization, and value aggregation, while deferring detailed multi-head implementation internals.
- Examples
- Uses the sentence “Wie geht es dir,” a four-token embedding, and a PyTorch attention-head implementation.
- What you can do after
- Apply it via scaled dot-product attention to normalize scores and aggregate value vectors.
Read how Build a Text-to-Image Generator (from Scratch) explains softmax →
Natural Language Processing in Action, Second Edition
Softmax is a normalized exponential function that converts a vector of scores into nonnegative values summing to one.
- How it is used
- Uses neural-network output layers to represent probabilities for mutually exclusive classification targets.
- How deep it goes
- Explains the normalized-exponential calculation, rank preservation, probability interpretation, and activation choice for multiclass versus multi-label tasks.
- Examples
- Uses three-dimensional score vectors and skip-gram predictions such as Monet targeting Claude or painted.
- What you can do after
- Evaluate/understand when a classification output should form one shared probability distribution.
Read how Natural Language Processing in Action, Second Edition explains softmax →
Inside Deep Learning: Math, Algorithms, Models
A function that turns a vector of scores into normalized weights over items or classes by exponentiating and renormalizing them.
- How it is used
- Converts classification scores into probabilities for interpreting predictions and calculating cross-entropy loss.
- How deep it goes
- Defines normalized class probabilities and demonstrates PyTorch application, without deriving the function’s formula.
- Examples
- Plots two-class probabilities as confidence contours for circle-and-cross data.
- What you can do after
- Apply via PyTorch’s `F.softmax` to convert model outputs into class probabilities and visualize confidence.
Read how Inside Deep Learning: Math, Algorithms, Models explains softmax →
Deep Learning with R, Third Edition
A softmax operation converts a vector of scores into nonnegative values that sum to one, typically representing class probabilities.
- How it is used
- Turns neural-network scores into class probabilities for classification and token probabilities for text generation.
- How deep it goes
- Explains probability normalization, demonstrates prediction output, and contrasts explicit softmax with logits without deriving the operation mathematically.
- Examples
- Uses a ten-class MNIST digit classifier and a transformer whose token outputs can be normalized from logits.
- What you can do after
- Apply it via Keras to produce class probabilities or configure cross-entropy to consume logits instead.
Read how Deep Learning with R, Third Edition explains softmax →
Hugging Face in Action
Softmax is a mathematical function that converts a vector of numbers into a probability distribution by exponentiating and normalizing its values.
- How it is used
- Converts model output logits into class or next-token probabilities for classification and transformer decoding.
- How deep it goes
- Defines exponentiation and normalization, notes that probabilities sum to one, and demonstrates application without deriving implementation internals.
- Examples
- Uses neural-network output values, next-token prediction, and CLIP labels for cat, dog, tiger, and train.
- What you can do after
- Apply it via PyTorch's softmax() to rank candidate labels after a CLIP model produces logits.
Machine Learning Algorithms in Depth
A function that converts a vector of scores into nonnegative values summing to one, typically interpreted as class probabilities.
- How it is used
- Converts multiclass classification scores into probabilities and constrains mixture-density-network weights to sum to one.
- How deep it goes
- Defines softmax through multiclass posteriors, but develops the accompanying from-scratch logistic-regression optimization for the binary case.
- Examples
- Uses MNIST digit classification and a synthetic-data Gaussian mixture density network; poses a temperature-parameter exercise.
- What you can do after
- Apply it via Keras/TensorFlow as a CNN classifier output or as constrained mixture weights.
Read how Machine Learning Algorithms in Depth explains softmax →
Data Without Labels
A softmax function converts multiple real-valued outputs into nonnegative values that sum to one, typically representing class probabilities.
- How it is used
- Uses softmax in neural-network output layers to classify inputs among multiple classes.
- How deep it goes
- Defines probability output and prediction selection, but does not detail the mathematical formula or implementation procedure.
- Examples
- Uses car-bike-truck and cat-dog image classification examples to show competing class probabilities.
- What you can do after
- Evaluate multiclass neural-network outputs by selecting the class with the highest softmax probability.
Sutskever's List
A function that converts a vector of scores into nonnegative values summing to one, interpreted as probabilities.
- How it is used
- Converts final-layer class scores into probabilities for choosing a classifier’s predicted category.
- How deep it goes
- Defines the score-to-probability transformation and shows its role in classification without detailing its computation.
- Examples
- Uses AlexNet’s 1,000 ImageNet category scores as the concrete classification example.
- What you can do after
- Understand how raw category scores become predicted probabilities and determine a classification.
Grokking AI Algorithms, Second Edition
The softmax function converts a vector of real-valued scores into nonnegative values that sum to one.
- How it is used
- Converts attention scores and language-model logits into normalized weights or token probabilities.
- How deep it goes
- Works through scaling, exponentiation, normalization, and numeric tables, but does not develop deeper theory.
- Examples
- Uses token attention tables and a 35-token vocabulary probability table to demonstrate normalization and prediction.
- What you can do after
- Evaluate attention relevance and token probabilities by following the softmax calculation from scores to normalized values.
Read how Grokking AI Algorithms, Second Edition explains softmax →
Understanding AI
Softmax is a function that converts a set of real-valued scores into nonnegative weights whose sum is one.
- How it is used
- Normalizes token-relevance scores into attention weights during attention processing.
- How deep it goes
- Defines the normalization role and illustrates its output, but does not derive the softmax formula or implementation.
- Examples
- Uses the sentence about unbelievable performance and assigns example weights to performance, crowd, and couldn't.
- What you can do after
- Evaluate how attention scores become a probability-like distribution over tokens.
Timeless Algorithms, The Foundational Ideas
The softmax function transforms a vector of real-valued scores into a probability distribution.
- How it is used
- Converts scores for mutually exclusive, collectively exhaustive outcomes into a probability distribution.
- How deep it goes
- Explains score-to-probability conversion and attention weighting through applications, without detailing the function’s mathematical form.
- Examples
- Uses class probabilities, next-word prediction, and attention examples involving pronoun resolution, semantic association, and factual recall.
- What you can do after
- Understand how scores become probabilities for classification and next-token selection, and how softmax weights contribute to attention.
Read how Timeless Algorithms, The Foundational Ideas explains softmax →
Also covered in
- Machines that ThinkIn early access (MEAP)
- Graph Neural Networks in Action
- Fight Fraud with Machine LearningIn early access (MEAP)
- Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI
What to know first
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- classifierA classifier is a predictive model that assigns an input to one or more discrete classes or categories.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- transformer architectureA transformer is a neural-network architecture that uses attention-based interactions and position-wise transformations to model relationships among elements of sequences or other structured inputs.
Related concepts
- probability distributionproduces · A probability distribution is a mathematical specification assigning probabilities or probability densities to the possible values or outcomes of a random variable.
- attentionused by · Attention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- logitused by · A logit is a real-valued quantity representing log-odds for a binary outcome or an unnormalized model score for a candidate outcome before probability normalization.
- temperatureconfigured by · A temperature parameter controls the spread of a probability distribution or the likelihood of selecting less-favored alternatives in stochastic sampling or search.
- reduction operationuses · A reduction operation combines multiple input values according to a rule to produce fewer results, often a single aggregate.
- fused Flash Attention kernelused by · The fused Flash Attention kernel is an example CUDA implementation that combines attention computations in one kernel.
Questions about softmax
What is softmax?
Softmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.
Which Manning books explain softmax?
CUDA for LLMs, Optimization Algorithms: AI techniques for design, planning, and control problems, Server-Side WebAssembly, Deep Learning with PyTorch, Second Edition, Build a Reasoning Model (From Scratch), Math and Architectures of Deep Learning, Rearchitecting LLMs, Grokking Machine Learning and 18 more.
What should I know before learning softmax?
Manning books that teach softmax usually introduce embedding, classifier, language model and transformer architecture first.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

























