Automatic differentiation, or autodiff, tracks the differentiable operations performed during a function call and automatically calculates the function’s exact gradient. It replaces manually deriving and implementing gradients, making complicated optimization objectives practical. section 6.5
In variational inference, autodiff supplies gradients for adjusting the mean and standard deviation of an approximating distribution while maximizing the ELBO. Sampling creates a difficulty: autodiff cannot directly see how those parameters affect an output through a random draw. section 6.5 section 6.15
Gradient information as flowing water
Imagine the code for the ELBO as a network of water pipes and gradient information as water flowing through them. Differentiable operations keep the water moving; a sampling operation acts like a blockage that prevents the gradient from reaching the distribution parameters. section 6.5







