liveBook · Manning

Gradient

As explained in Grokking Bayes

A gradient is the vector of partial derivatives that indicates the direction of greatest local increase of a differentiable function.

A gradient indicates how a function changes near the current parameter values. Optimization uses it to choose a direction: move downhill to minimize a function or uphill to maximize one. section 6.5

Gradient-based optimization proceeds iteratively:

  • Compute the gradient of the objective at the current location. section 6.5
  • Adjust the parameters using the gradient direction. section 6.5
  • Repeat until further steps no longer find a lower value and optimization converges. section 6.5

Optimizing a variational approximation

Variational inference adjusts the mean μ and standard deviation σ of a Normal approximating distribution. Because the ELBO is maximized, the update moves opposite the usual descent direction; this is gradient ascent. section 6.5

How Grokking Bayes covers it

How deep it goes
Defines the direction of steepest decrease and gives an iterative update-and-convergence procedure, without developing technical details.
Examples
Uses μ and σ of a Normal approximating distribution and the ELBO as the optimization objective.
What you can do after
Apply gradient descent or ascent to iteratively optimize parameters using objective-function gradients.
How it connects
Connects gradients with gradient descent, gradient ascent, ELBO, variational inference, and parameter optimization.

Also explained in

Compare how 27 books explain gradient →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Bayes. The section links point to the parts of the book it draws on.