liveBook · Manning

Automatic differentiation

As explained in Grokking Bayes

Automatic differentiation is a technique that computationally tracks differentiable operations in a function to calculate its exact derivatives.

Automatic differentiation, or autodiff, tracks the differentiable operations performed during a function call and automatically calculates the function’s exact gradient. It replaces manually deriving and implementing gradients, making complicated optimization objectives practical. section 6.5

In variational inference, autodiff supplies gradients for adjusting the mean and standard deviation of an approximating distribution while maximizing the ELBO. Sampling creates a difficulty: autodiff cannot directly see how those parameters affect an output through a random draw. section 6.5 section 6.15

Gradient information as flowing water

Imagine the code for the ELBO as a network of water pipes and gradient information as water flowing through them. Differentiable operations keep the water moving; a sampling operation acts like a blockage that prevents the gradient from reaching the distribution parameters. section 6.5

How Grokking Bayes covers it

How deep it goes
Explains operation tracking, gradient flow, and sampling’s obstruction but stops short of implementation details.
Examples
Uses ELBO optimization and a water-pipe analogy to illustrate gradients passing through differentiable operations and stopping at sampling.
What it assumes
Assumes familiarity with gradient descent, ELBO optimization, and differentiable operations; highlights sampling as a gradient obstacle.
What you can do after
Apply it via gradient descent to optimize differentiable objectives without manually deriving their gradients.
How it connects
Connects automatic differentiation with gradient descent, variational inference, ELBO optimization, sampling, and the reparameterization trick.

Also explained in

Compare how 8 books explain automatic differentiation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Bayes. The section links point to the parts of the book it draws on.