liveBook · Manning

Automatic differentiation

Also called autograd, autodiff, autodifferentiation

Automatic differentiation is a computational technique that evaluates derivatives of a program by systematically applying the chain rule to its elementary operations.

How 8 Manning books explain it

Inside Deep Learning: Math, Algorithms, Models

A mechanism for computing derivatives of model computations automatically, used here to support training of networks with registered parameters.

How it is used
Computes gradients that guide loss-minimizing updates to neural network parameters.
How deep it goes
Walks through tracked tensor operations, `.backward()`, and an iterative update; leaves derivative computation itself to PyTorch.
Examples
Uses `f(x) = (x − 2)²` at `x = −3.5`, then iterates toward its minimum in a PyTorch loop.
What you can do after
Apply it via PyTorch to compute gradients and use them in loss-minimizing parameter updates.

Read how Inside Deep Learning: Math, Algorithms, Models explains automatic differentiation →

Deep Learning with JAX

A technique for computing exact derivatives of code-expressed computations by systematically applying differentiation rules to the program structure.

How it is used
Supports gradient-based neural-network training and differentiable numerical programs within JAX's composable transformation system.
How deep it goes
Descends from practical grad() usage to forward and reverse modes, JVPs, VJPs, Jacobians, and Hessians.
Examples
Uses a scalar polynomial-rational function, a hand-derived derivative, JAX grad(), and neural-network loss gradients.
What you can do after
Apply it via JAX grad(), value_and_grad(), jacfwd(), jacrev(), hessian(), jvp(), and vjp() to obtain derivatives.

Read how Deep Learning with JAX explains automatic differentiation →

Deep Learning with R, Third Edition

Automatic differentiation computes exact derivatives of programs composed of differentiable operations by systematically applying the chain rule.

How it is used
Computes loss gradients for neural network weight updates without hand-writing derivative logic.
How deep it goes
Explains the chain rule and computation graphs, then implements a training step with GradientTape rather than backpropagation from scratch.
Examples
Uses a linear tensor expression and a per-batch neural network training step to demonstrate gradient computation.
What you can do after
Apply it via TensorFlow GradientTape to compute loss gradients for a Keras training step.

Read how Deep Learning with R, Third Edition explains automatic differentiation →

Build AI Drug Discovery Pipelines

Automatic differentiation is the systematic computation of exact derivatives of programs by applying the chain rule to their elementary operations.

How it is used
Trains neural networks by backpropagating loss gradients through computational graphs.
How deep it goes
Explains graph structure, tracked parameters, and backward gradient computation, but stops short of derivative internals.
Examples
Uses logistic regression with feature, weight, bias, sigmoid, binary-cross-entropy, and loss.backward() operations.
What you can do after
Apply it via PyTorch to compute parameter gradients from a model loss.

Read how Build AI Drug Discovery Pipelines explains automatic differentiation →

Deep Learning with PyTorch, Second Edition

A technique for automatically computing derivatives needed for training neural networks.

How it is used
Supports gradient-based parameter estimation by calculating how model loss changes with its parameters.
How deep it goes
Explains gradient tracking and computation graphs, then applies backward and parameter updates in a training loop.
Examples
Uses thermometer calibration with a linear model, mean squared loss, and a repeated parameter-update loop.
What you can do after
Apply it via PyTorch autograd to compute gradients and update model parameters.

Read how Deep Learning with PyTorch, Second Edition explains automatic differentiation →

Deep Learning with Python, Third Edition

A mechanism that programmatically computes exact derivatives of recorded operations by applying the chain rule to a computation graph.

How it is used
Provides gradients for differentiable functions in neural network computation and deep learning frameworks.
How deep it goes
Explains the chain rule and computation graphs as the basis for backpropagation, then identifies automatic differentiation as a framework feature.
Examples
Uses a chain of composed functions and a two-layer neural network loss calculation to illustrate differentiable operations.
What you can do after
Evaluate and understand how a computation graph supports automatic gradient calculation.

Read how Deep Learning with Python, Third Edition explains automatic differentiation →

Math and Architectures of Deep Learning

A method for mechanically computing derivatives of functions by tracing elementary operations.

How it is used
Scales gradient computation from simple formulas to nonlinear models with millions of parameters.
How deep it goes
Explains requires_grad, computation graphs, backward(), and optimizer updates, but does not derive Autograd internals.
Examples
Uses a noisy linear model and a demand-prediction neural network trained with mean squared error.
What you can do after
Apply it via PyTorch Autograd to compute gradients and train linear or neural-network models.

Read how Math and Architectures of Deep Learning explains automatic differentiation →

Grokking Bayes

Automatic differentiation is a technique that computationally tracks differentiable operations in a function to calculate its exact derivatives.

How it is used
Supports gradient-based optimization for complicated objectives in modern machine learning pipelines.
How deep it goes
Explains operation tracking, gradient flow, and sampling’s obstruction but stops short of implementation details.
Examples
Uses ELBO optimization and a water-pipe analogy to illustrate gradients passing through differentiable operations and stopping at sampling.
What you can do after
Apply it via gradient descent to optimize differentiable objectives without manually deriving their gradients.

Read how Grokking Bayes explains automatic differentiation →

Also covered in

What to know first

Related concepts

Questions about automatic differentiation

What is automatic differentiation?

Automatic differentiation is a computational technique that evaluates derivatives of a program by systematically applying the chain rule to its elementary operations.

Which Manning books explain automatic differentiation?

Inside Deep Learning: Math, Algorithms, Models, Deep Learning with JAX, Deep Learning with R, Third Edition, Build AI Drug Discovery Pipelines, Deep Learning with PyTorch, Second Edition, Deep Learning with Python, Third Edition, Math and Architectures of Deep Learning and Grokking Bayes.

What should I know before learning automatic differentiation?

Manning books that teach automatic differentiation usually introduce gradient, optimization, language model and chain rule first.

What is automatic differentiation also called?

Automatic differentiation is also called autograd, autodiff or autodifferentiation.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.