liveBook · Manning

Softmax

As explained in GPU Programming with Triton

Softmax normalization exponentiates scores and divides by their sum to produce nonnegative values that sum to one.

Softmax maps unconstrained scores (logits) to nonnegative values that sum to one. It is used for class confidence and query-key similarities; each output depends on the full vector through the denominator. section 4.2

For stable evaluation, subtract the vector maximum m before exponentiating: yᵢ = exp(xᵢ − m) / Σⱼ exp(xⱼ − m). This preserves the probabilities and keeps the largest exponential at 1; very negative entries may underflow but contribute negligible mass. section 4.2.1

An unfused one-dimensional Triton implementation follows four steps:

Separate kernels and host-side reduction loops incur launches and global-memory round-trips for partial reductions and intermediate values. For row-wise softmax, map the program ID to the row and add a row stride. section 4.2.2

A stable shift

For x = [1000, 1001, 1002], subtracting m = 1002 gives [-2, -1, 0]. The exponentials [e⁻², e⁻¹, 1] stay within range and yield the same probabilities as the unshifted formula. section 4.2.1

Figure 4.7 (Left) Dynamic range and precision comparison between FP32 and FP16. FP32 covers a far wider range with finer spacing between representable values. (Right) Bit-level breakdown showing how both formats encode the same number, with FP16 producing a less precise result due to its shorter mantissa.
CH04 F07 Fartale

Compare FP16 and FP32 dynamic range and precision to see why lower-precision softmax is more vulnerable to overflow and instability. section 4.2.1

How GPU Programming with Triton covers it

How deep it goes
Explains maximum-shift stability and outlines an unfused Triton pipeline with global reductions and intermediate materialization.
Examples
Works through logits [1000, 1001, 1002] and contrasts FP16 with FP32 range and precision.
What you can do after
Evaluate numerical stability and memory costs by tracing maximum, exponentiation, sum, and normalization.
How it connects
Connects softmax to reductions, attention mechanisms, and floating-point numerical stability.

Also explained in

Compare how 26 books explain softmax →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.