Softmax

As explained in Machine Learning Algorithms in Depth

A function that converts a vector of scores into nonnegative values summing to one, typically interpreted as class probabilities.

Softmax turns K real-valued scores into nonnegative values that sum to one. For scores a₁,…,aₖ, the output for class k is exp(aₖ) divided by the sum of exp(aⱼ) over all classes j. The resulting vector can be interpreted as a distribution over classes. section 5.4

In probabilistic classification, the target is p(Cₖ|x): the probability of class Cₖ given input x. Bayes’ rule supplies a score aₖ = ln[p(x|Cₖ)p(Cₖ)] for each class. Exponentiating and normalizing those scores gives the multiclass posterior. This connects softmax to both the class-conditional density and the class prior. section 5.4

(5.21)

The multiclass posterior equation shows how the class scores are normalized. section 5.4

The same normalization serves different outputs:

• A softmax classifier follows the dense layer of a LeNet-style CNN for MNIST handwritten-digit classification. section 10.2.1
• Softmax activations make the mixture weights of a Gaussian mixture density network sum to one; exponential activations separately keep its variances positive. section 11.2.1

Temperature softmax is a further practice question: examine how changing its temperature parameter T changes the output. The binary logistic-regression implementation is not, by itself, an implementation of a multiclass softmax classifier. section 7.7 section 5.4

How Machine Learning Algorithms in Depth covers it

How deep it goes

Defines softmax through multiclass posteriors, but develops the accompanying from-scratch logistic-regression optimization for the binary case.

Examples

Uses MNIST digit classification and a synthetic-data Gaussian mixture density network; poses a temperature-parameter exercise.

What it assumes

Assumes basic probability, introducing class-posterior notation through Bayes’ rule before the multiclass transformation.

What you can do after

Apply it via Keras/TensorFlow as a CNN classifier output or as constrained mixture weights.

How it connects

Links Bayes’ rule and logistic regression to CNN classification, Gaussian mixture models, and temperature softmax.

liveBookmanning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsSoftmax is a function that converts a vector of real-valued scores into nonnegative class probabilities that sum to one.
• 

  liveBookmanning.com
  Server-Side WebAssemblySoftmax is a mathematical transformation that converts a vector of scores into nonnegative probabilities summing to one.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA normalization function that converts a vector of scores into a probability distribution by exponentiating and renormalizing the values.
• 

  liveBookmanning.com
  Build a Reasoning Model (From Scratch)A normalization function that converts logits into a probability distribution over possible next tokens.
• 

  liveBookmanning.com
  Math and Architectures of Deep LearningA smooth function that converts an unnormalized score vector into a probability-like distribution over classes.
• 

  liveBookmanning.com
  Grokking Machine LearningA function that converts multiple real-valued scores into positive values whose normalized values sum to one and represent class probabilities.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchAn activation that converts raw scores into a probability distribution over multiple classes or positions.
• 

  liveBookmanning.com
  Transformers in ActionThe softmax function converts a vector of scores into nonnegative values that sum to one.
• 

  liveBookmanning.com
  Build a Text-to-Image Generator (from Scratch)A function that converts a vector of real-valued scores into nonnegative values that sum to one.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionSoftmax is a normalized exponential function that converts a vector of scores into nonnegative values summing to one.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsA function that turns a vector of scores into normalized weights over items or classes by exponentiating and renormalizing them.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA softmax operation converts a vector of scores into nonnegative values that sum to one, typically representing class probabilities.
• 

  liveBookmanning.com
  Hugging Face in ActionSoftmax is a mathematical function that converts a vector of numbers into a probability distribution by exponentiating and normalizing its values.
• 

  liveBookmanning.com
  Data Without LabelsA softmax function converts multiple real-valued outputs into nonnegative values that sum to one, typically representing class probabilities.
• 

  liveBookmanning.com
  Sutskever's ListA function that converts a vector of scores into nonnegative values summing to one, interpreted as probabilities.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionThe softmax function converts a vector of real-valued scores into nonnegative values that sum to one.

Compare how 17 books explain softmax →

Related concepts

• probability distributionproduces · A probability distribution is a mathematical specification assigning probabilities to the possible values or outcomes of a random variable, with total probability one.
• temperatureused by · A temperature parameter adjusts the concentration of a probability distribution, affecting sampling randomness or the likelihood of accepting less favorable transitions.
• reduction operationuses · A reduction operation combines multiple input values or partial results according to a combining rule to produce fewer results, often a single aggregate.
• probability theoryproduces · Probability theory is the branch of mathematics that formalizes uncertainty and analyzes random events and outcomes using probability measures.
• attentionused by · Attention is a neural-network mechanism that computes input-dependent weights to combine information according to its relevance to a computation.
• attention weightproduces · An attention weight is a computed coefficient, often normalized, that determines how strongly an input element or value contributes to an attention-based representation.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.