Stable reformulation changes a computation to reduce overflow, underflow, or cancellation where numerical errors can grow. It is especially useful when reductions feed sensitive nonlinear operations such as softmax. section 4.1.1
Two examples show how the computation can be reshaped:
- For softmax, subtract each row’s maximum before exponentiating, shifting values into a safer range. section 4.1.1
- For products of positive values, sum their logarithms and exponentiate the result; this can prevent product overflow or underflow, but adds logarithm and exponential costs. section 4.1.1
\[\prod_{i=1}^{n} x_i = \exp\!\left(\sum_{i=1}^{n} \log x_i\right)\]
The product-to-sum identity expresses the log-space reformulation. section 4.1.1
