Softmax turns a vector of scores into nonnegative values that sum to one. Applied to a neural network’s output layer, it produces a probability distribution over mutually exclusive classes, making the largest score the most likely class. section 6.3.2
For output score zᵢ, softmax computes exp(zᵢ) divided by the sum of exp(zⱼ) across all k outputs. This preserves the scores’ rank order while scaling them into probabilities. section 6.3.2
Choose the output activation according to the prediction task:
- Use softmax for multiclass classification, where exactly one mutually exclusive category should describe each instance. section 8.2.4
- Use element-wise sigmoid instead for multi-label tagging, where several independent labels can apply at once. section 8.2.4
Predicting a neighboring word
In skip-gram Word2Vec training, the input token Monet can produce output scores for possible target words such as Claude or painted. Softmax converts those scores into a distribution whose entries sum to approximately 1, allowing the network to represent its predicted target probabilities. section 6.3.2
Equation 6.3 Softmax
Inspect the normalized-exponential softmax equation alongside example vectors before and after the transformation. section 6.3.2
