liveBook · Manning

Numerical stability

As explained in GPU Programming with Triton

The property of a numerical computation in which small input or rounding errors do not cause disproportionately large errors in its results.

Softmax turns logits into non-negative values that sum to one, but exponentials can overflow for moderately large inputs. A maximum reduction helps prevent overflow; computing the denominator also requires a sum reduction across the vector. section 4.2

Softmax Activation Function
\[\mathbf{y} = \text{softmax}(\mathbf{x}) y_i = \frac{\exp(x_i)}{\sum_{j=0}^{N-1} \exp(x_j)} \quad \text{for } i = 0, 1, \ldots, N-1\]

The softmax formula shows where exponentials and the vector-wide denominator create numerical and reduction challenges. section 4.2

How GPU Programming with Triton covers it

How deep it goes
Explains why softmax uses a maximum reduction alongside its denominator sum, without detailing a complete implementation.
Examples
Uses class-confidence logits and query-key similarities as softmax inputs.
What you can do after
Evaluate why softmax needs a maximum reduction to prevent exponential overflow.
How it connects
Connects numerical stability with softmax, reductions, and attention.

Also explained in

Compare how 3 books explain numerical stability →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.