liveBook · Manning

Stable reformulation

As explained in GPU Programming with Triton

A stable reformulation rewrites a numerical computation to reduce amplification of overflow, underflow, or cancellation errors.

Stable reformulation changes a computation to reduce overflow, underflow, or cancellation where numerical errors can grow. It is especially useful when reductions feed sensitive nonlinear operations such as softmax. section 4.1.1

Two examples show how the computation can be reshaped:

  • For softmax, subtract each row’s maximum before exponentiating, shifting values into a safer range. section 4.1.1
  • For products of positive values, sum their logarithms and exponentiate the result; this can prevent product overflow or underflow, but adds logarithm and exponential costs. section 4.1.1
\[\prod_{i=1}^{n} x_i = \exp\!\left(\sum_{i=1}^{n} \log x_i\right)\]

The product-to-sum identity expresses the log-space reformulation. section 4.1.1

How GPU Programming with Triton covers it

How deep it goes
Explains maximum subtraction and log-space transformation, but defers the softmax implementation.
Examples
Uses row-maximum subtraction before exponentiation and converts products of positive values into sums of logarithms.
What you can do after
Apply maximum subtraction before exponentiation or transform positive-value products into log-space sums.
How it connects
Connects stable reformulation to softmax, reductions, floating-point associativity, and log-space transformations.

This explainer was generated by Manning's knowledge graph from the text of GPU Programming with Triton. The section links point to the parts of the book it draws on.