Softmax maps unconstrained scores (logits) to nonnegative values that sum to one. It is used for class confidence and query-key similarities; each output depends on the full vector through the denominator. section 4.2
For stable evaluation, subtract the vector maximum m before exponentiating: yᵢ = exp(xᵢ − m) / Σⱼ exp(xⱼ − m). This preserves the probabilities and keeps the largest exponential at 1; very negative entries may underflow but contribute negligible mass. section 4.2.1
An unfused one-dimensional Triton implementation follows four steps:
- Reduce the input to its global maximum. section 4.2.2
- Materialize shifted exponentials in global memory. section 4.2.2
- Reduce the exponentials to their sum, then normalize. section 4.2.2
Separate kernels and host-side reduction loops incur launches and global-memory round-trips for partial reductions and intermediate values. For row-wise softmax, map the program ID to the row and add a row stride. section 4.2.2
A stable shift
For x = [1000, 1001, 1002], subtracting m = 1002 gives [-2, -1, 0]. The exponentials [e⁻², e⁻¹, 1] stay within range and yield the same probabilities as the unshifted formula. section 4.2.1
Figure 4.7 (Left) Dynamic range and precision comparison between FP32 and FP16. FP32 covers a far wider range with finer spacing between representable values. (Right) Bit-level breakdown showing how both formats encode the same number, with FP16 producing a less precise result due to its shorter mantissa.
Compare FP16 and FP32 dynamic range and precision to see why lower-precision softmax is more vulnerable to overflow and instability. section 4.2.1

























