A gradient indicates how a function changes near the current parameter values. Optimization uses it to choose a direction: move downhill to minimize a function or uphill to maximize one. section 6.5
Gradient-based optimization proceeds iteratively:
- Compute the gradient of the objective at the current location. section 6.5
- Adjust the parameters using the gradient direction. section 6.5
- Repeat until further steps no longer find a lower value and optimization converges. section 6.5
Optimizing a variational approximation
Variational inference adjusts the mean μ and standard deviation σ of a Normal approximating distribution. Because the ELBO is maximized, the update moves opposite the usual descent direction; this is gradient ascent. section 6.5


























