liveBook · Manning

Softmax

Softmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.

How 26 Manning books explain it

CUDA for LLMs

Softmax is a function that converts real-valued scores into nonnegative values that sum to one.

How it is used
Uses softmax to normalize classifier logits and transformer attention scores while motivating efficient GPU reductions.
How deep it goes
Develops stable formulas and naive CUDA kernels into online, shared-memory, and tiled attention implementations.
Examples
Works through four logits numerically and implements row-wise CUDA kernels with running normalization statistics.
What you can do after
Builds softmax from scratch using stable CPU and CUDA implementations, then applies online and shared-memory optimizations.

Read how CUDA for LLMs explains softmax →

Optimization Algorithms: AI techniques for design, planning, and control problems

Softmax is a function that converts a vector of real-valued scores into nonnegative class probabilities that sum to one.

How it is used
Converts classifier outputs to probabilities and normalizes attention scores for graph and pointer networks.
How deep it goes
Shows exponentiation and row-wise normalization in NumPy, then uses the resulting probabilities in a loss calculation.
Examples
Uses penguin species classification and convex hull point selection to illustrate probability outputs.
What you can do after
Apply it in NumPy to obtain class probabilities and calculate negative log-likelihood.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains softmax →

Server-Side WebAssembly

Softmax is a mathematical transformation that converts a vector of scores into nonnegative probabilities summing to one.

How it is used
Interprets ML inference outputs as class probabilities or next-token sampling weights.
How deep it goes
Works through exponentiation, summation, normalization, ranking, and weighted sampling without discussing mathematical derivation.
Examples
Uses a SqueezeNet Persian-cat prediction and a “Once upon a time” story-generation loop.
What you can do after
Apply it via wasi-nn inference outputs to rank image labels or sample generated tokens.

Read how Server-Side WebAssembly explains softmax →

Deep Learning with PyTorch, Second Edition

A normalization function that converts a vector of scores into a probability distribution by exponentiating and renormalizing the values.

How it is used
Converts classifier scores into probabilities and attention scores into normalized weights.
How deep it goes
Explains exponentiation and normalization with worked vectors, then demonstrates dimension selection and model integration.
Examples
Uses score vectors, a bird-versus-airplane classifier, and a dot product attention implementation.
What you can do after
Apply it via PyTorch's nn.Softmax or F.softmax to normalize scores along a chosen dimension.

Read how Deep Learning with PyTorch, Second Edition explains softmax →

Build a Reasoning Model (From Scratch)

A normalization function that converts logits into a probability distribution over possible next tokens.

How it is used
Converts temperature-scaled next-token logits into probabilities used by the sampling step.
How deep it goes
Explains normalization, temperature effects, plotting, and PyTorch usage, then stops before deeper mathematical derivation.
Examples
Uses the " Berlin" token, probability-sum checks, plots, and repeated multinomial sampling.
What you can do after
Apply it via PyTorch to normalize logits and feed probabilities into token sampling.

Read how Build a Reasoning Model (From Scratch) explains softmax →

Math and Architectures of Deep Learning

A smooth function that converts an unnormalized score vector into a probability-like distribution over classes.

How it is used
Converts classifier score vectors into class probabilities for prediction and cross-entropy-based training.
How deep it goes
Defines score normalization, probability properties, confidence interpretation, and smooth argmax approximation, then connects it to PyTorch training behavior.
Examples
Uses cat-dog-airplane-automobile scores, a three-class height-and-weight classifier, MNIST LeNet, and Faster R-CNN regions.
What you can do after
Apply it via PyTorch to convert prediction scores into probabilities and select a class with argmax.

Read how Math and Architectures of Deep Learning explains softmax →

Rearchitecting LLMs

Softmax converts a vector of scores into nonnegative probabilities that sum to one.

How it is used
Converts attention scores into value weights and teacher logits into soft labels for distillation.
How deep it goes
Explains normalization and a PyTorch distillation workflow but does not derive the softmax formula.
Examples
Uses logits [2.0, 1.0, 0.1] and an attention query for “spring” to illustrate normalization.
What you can do after
Apply it via PyTorch alongside log_softmax and KL divergence to compute a distillation logits loss.

Read how Rearchitecting LLMs explains softmax →

Grokking Machine Learning

A function that converts multiple real-valued scores into positive values whose normalized values sum to one and represent class probabilities.

How it is used
Produces probability distributions when a classifier must choose among more than two classes.
How deep it goes
Explains the motivation, exponential normalization, worked scores, and Keras implementation without deriving deeper optimization details.
Examples
Uses dog, cat, and bird scores, a two-class circular dataset, and handwritten MNIST digit classification.
What you can do after
Apply softmax in Keras to create multi-class outputs and select predictions from their highest probabilities.

Read how Grokking Machine Learning explains softmax →

Learn Generative AI with PyTorch

An activation that converts raw scores into a probability distribution over multiple classes or positions.

How it is used
Converts classifier logits, attention scores, and language-model logits into normalized probabilities or weights.
How deep it goes
Explains normalization through classifier and attention workflows, without deriving the softmax formula.
Examples
Uses a ten-class clothing classifier, a four-token attention example, and GPT-2XL vocabulary logits.
What you can do after
Apply it via PyTorch loss functions when training a multicategory classifier and interpret normalized attention weights.

Read how Learn Generative AI with PyTorch explains softmax →

Applied Reinforcement Learning

Softmax converts a vector of real-valued scores into nonnegative probabilities that sum to one.

How it is used
Maps policy logits to discrete-action probabilities and attention scores to weights.
How deep it goes
Defines the score normalization and works through a numerical action example and attention’s saturation concern.
Examples
Uses factory production logits and temperature-zone query-key scores as examples.
What you can do after
Apply softmax to logits or similarity scores to obtain action probabilities or normalized attention weights.

Read how Applied Reinforcement Learning explains softmax →

Grokking Machine Learning, Second Edition

The softmax function converts a vector of real-valued scores into positive values that sum to one, yielding a probability distribution over classes.

How it is used
Turns competing classifier scores into a probability distribution for multiclass prediction.
How deep it goes
Explains exponential transformation, normalization, worked values, and neural-network placement without deriving training internals.
Examples
Uses dog, cat, and bird scores, plus a Keras network classifying a two-feature dataset with two softmax outputs.
What you can do after
Apply it via Keras as a multiclass output activation and interpret normalized class probabilities.

Read how Grokking Machine Learning, Second Edition explains softmax →

Build a DeepSeek Model (From Scratch)

Softmax is a function that converts a vector of scores into nonnegative values summing to one.

How it is used
Converts scores into weights for attention, token prediction, expert routing, and knowledge distillation.
How deep it goes
Works through exponentiation and normalization numerically, and explains masking and temperature scaling in practical settings.
Examples
Uses four-class image logits at three temperatures and top-k expert scores to demonstrate normalization.
What you can do after
Evaluate how masking and temperature affect normalized weights in attention, expert routing, and distillation.

Read how Build a DeepSeek Model (From Scratch) explains softmax →

Build AI Drug Discovery Pipelines

Softmax is a function that converts a vector of scores into nonnegative values summing to one.

How it is used
Deploys probabilities for multiclass CYP inhibition and next-token selection during SMILES molecule generation.
How deep it goes
Explains class-specific weights, logits, probability conversion, prediction, and cross-entropy, while flagging the single-class assumption.
Examples
Uses five CYP inhibition classes, SMILES vocabulary prediction, and temperature-based molecule generation as concrete applications.
What you can do after
Apply it via multiclass models and molecular decoders to interpret class probabilities or sample the next SMILES character.

Read how Build AI Drug Discovery Pipelines explains softmax →

GPU Programming with Triton

Softmax normalization exponentiates scores and divides by their sum to produce nonnegative values that sum to one.

How it is used
Uses softmax as a chained-reduction case study for classification logits and attention similarities.
How deep it goes
Explains maximum-shift stability and outlines an unfused Triton pipeline with global reductions and intermediate materialization.
Examples
Works through logits [1000, 1001, 1002] and contrasts FP16 with FP32 range and precision.
What you can do after
Evaluate numerical stability and memory costs by tracing maximum, exponentiation, sum, and normalization.

Read how GPU Programming with Triton explains softmax →

Transformers in Action

The softmax function converts a vector of scores into nonnegative values that sum to one.

How it is used
Uses softmax to produce token-sampling probabilities and normalize attention scores.
How deep it goes
Explains temperature effects and the attention-weighted sum, with library calls rather than a derivation of softmax.
Examples
Demonstrates generation at temperature 0.7 and attention over input embeddings using query, key, and value matrices.
What you can do after
Apply it via Transformers generation settings to adjust sampling randomness.

Read how Transformers in Action explains softmax →

Build a Text-to-Image Generator (from Scratch)

A function that converts a vector of real-valued scores into nonnegative values that sum to one.

How it is used
Converts scaled query-key scores into attention weights that determine how tokens combine value vectors.
How deep it goes
Explains scoring, scaling, normalization, and value aggregation, while deferring detailed multi-head implementation internals.
Examples
Uses the sentence “Wie geht es dir,” a four-token embedding, and a PyTorch attention-head implementation.
What you can do after
Apply it via scaled dot-product attention to normalize scores and aggregate value vectors.

Read how Build a Text-to-Image Generator (from Scratch) explains softmax →

Natural Language Processing in Action, Second Edition

Softmax is a normalized exponential function that converts a vector of scores into nonnegative values summing to one.

How it is used
Uses neural-network output layers to represent probabilities for mutually exclusive classification targets.
How deep it goes
Explains the normalized-exponential calculation, rank preservation, probability interpretation, and activation choice for multiclass versus multi-label tasks.
Examples
Uses three-dimensional score vectors and skip-gram predictions such as Monet targeting Claude or painted.
What you can do after
Evaluate/understand when a classification output should form one shared probability distribution.

Read how Natural Language Processing in Action, Second Edition explains softmax →

Inside Deep Learning: Math, Algorithms, Models

A function that turns a vector of scores into normalized weights over items or classes by exponentiating and renormalizing them.

How it is used
Converts classification scores into probabilities for interpreting predictions and calculating cross-entropy loss.
How deep it goes
Defines normalized class probabilities and demonstrates PyTorch application, without deriving the function’s formula.
Examples
Plots two-class probabilities as confidence contours for circle-and-cross data.
What you can do after
Apply via PyTorch’s `F.softmax` to convert model outputs into class probabilities and visualize confidence.

Read how Inside Deep Learning: Math, Algorithms, Models explains softmax →

Deep Learning with R, Third Edition

A softmax operation converts a vector of scores into nonnegative values that sum to one, typically representing class probabilities.

How it is used
Turns neural-network scores into class probabilities for classification and token probabilities for text generation.
How deep it goes
Explains probability normalization, demonstrates prediction output, and contrasts explicit softmax with logits without deriving the operation mathematically.
Examples
Uses a ten-class MNIST digit classifier and a transformer whose token outputs can be normalized from logits.
What you can do after
Apply it via Keras to produce class probabilities or configure cross-entropy to consume logits instead.

Read how Deep Learning with R, Third Edition explains softmax →

Hugging Face in Action

Softmax is a mathematical function that converts a vector of numbers into a probability distribution by exponentiating and normalizing its values.

How it is used
Converts model output logits into class or next-token probabilities for classification and transformer decoding.
How deep it goes
Defines exponentiation and normalization, notes that probabilities sum to one, and demonstrates application without deriving implementation internals.
Examples
Uses neural-network output values, next-token prediction, and CLIP labels for cat, dog, tiger, and train.
What you can do after
Apply it via PyTorch's softmax() to rank candidate labels after a CLIP model produces logits.

Read how Hugging Face in Action explains softmax →

Machine Learning Algorithms in Depth

A function that converts a vector of scores into nonnegative values summing to one, typically interpreted as class probabilities.

How it is used
Converts multiclass classification scores into probabilities and constrains mixture-density-network weights to sum to one.
How deep it goes
Defines softmax through multiclass posteriors, but develops the accompanying from-scratch logistic-regression optimization for the binary case.
Examples
Uses MNIST digit classification and a synthetic-data Gaussian mixture density network; poses a temperature-parameter exercise.
What you can do after
Apply it via Keras/TensorFlow as a CNN classifier output or as constrained mixture weights.

Read how Machine Learning Algorithms in Depth explains softmax →

Data Without Labels

A softmax function converts multiple real-valued outputs into nonnegative values that sum to one, typically representing class probabilities.

How it is used
Uses softmax in neural-network output layers to classify inputs among multiple classes.
How deep it goes
Defines probability output and prediction selection, but does not detail the mathematical formula or implementation procedure.
Examples
Uses car-bike-truck and cat-dog image classification examples to show competing class probabilities.
What you can do after
Evaluate multiclass neural-network outputs by selecting the class with the highest softmax probability.

Read how Data Without Labels explains softmax →

Sutskever's List

A function that converts a vector of scores into nonnegative values summing to one, interpreted as probabilities.

How it is used
Converts final-layer class scores into probabilities for choosing a classifier’s predicted category.
How deep it goes
Defines the score-to-probability transformation and shows its role in classification without detailing its computation.
Examples
Uses AlexNet’s 1,000 ImageNet category scores as the concrete classification example.
What you can do after
Understand how raw category scores become predicted probabilities and determine a classification.

Read how Sutskever's List explains softmax →

Grokking AI Algorithms, Second Edition

The softmax function converts a vector of real-valued scores into nonnegative values that sum to one.

How it is used
Converts attention scores and language-model logits into normalized weights or token probabilities.
How deep it goes
Works through scaling, exponentiation, normalization, and numeric tables, but does not develop deeper theory.
Examples
Uses token attention tables and a 35-token vocabulary probability table to demonstrate normalization and prediction.
What you can do after
Evaluate attention relevance and token probabilities by following the softmax calculation from scores to normalized values.

Read how Grokking AI Algorithms, Second Edition explains softmax →

Understanding AI

Softmax is a function that converts a set of real-valued scores into nonnegative weights whose sum is one.

How it is used
Normalizes token-relevance scores into attention weights during attention processing.
How deep it goes
Defines the normalization role and illustrates its output, but does not derive the softmax formula or implementation.
Examples
Uses the sentence about unbelievable performance and assigns example weights to performance, crowd, and couldn't.
What you can do after
Evaluate how attention scores become a probability-like distribution over tokens.

Read how Understanding AI explains softmax →

Timeless Algorithms, The Foundational Ideas

The softmax function transforms a vector of real-valued scores into a probability distribution.

How it is used
Converts scores for mutually exclusive, collectively exhaustive outcomes into a probability distribution.
How deep it goes
Explains score-to-probability conversion and attention weighting through applications, without detailing the function’s mathematical form.
Examples
Uses class probabilities, next-word prediction, and attention examples involving pronoun resolution, semantic association, and factual recall.
What you can do after
Understand how scores become probabilities for classification and next-token selection, and how softmax weights contribute to attention.

Read how Timeless Algorithms, The Foundational Ideas explains softmax →

Also covered in

What to know first

Related concepts

Questions about softmax

What is softmax?

Softmax is a function that exponentiates each value in a score vector and normalizes the results to produce positive values summing to one, forming a probability distribution.

Which Manning books explain softmax?

CUDA for LLMs, Optimization Algorithms: AI techniques for design, planning, and control problems, Server-Side WebAssembly, Deep Learning with PyTorch, Second Edition, Build a Reasoning Model (From Scratch), Math and Architectures of Deep Learning, Rearchitecting LLMs, Grokking Machine Learning and 18 more.

What should I know before learning softmax?

Manning books that teach softmax usually introduce embedding, classifier, language model and transformer architecture first.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.