Automatic differentiation
Automatic differentiation is a computational technique that evaluates derivatives of a program by systematically applying the chain rule to its elementary operations.
Inside Deep Learning: Math, Algorithms, Models
A mechanism for computing derivatives of model computations automatically, used here to support training of networks with registered parameters.
- How it is used
- Computes gradients that guide loss-minimizing updates to neural network parameters.
- How deep it goes
- Walks through tracked tensor operations, `.backward()`, and an iterative update; leaves derivative computation itself to PyTorch.
- Examples
- Uses `f(x) = (x − 2)²` at `x = −3.5`, then iterates toward its minimum in a PyTorch loop.
- What you can do after
- Apply it via PyTorch to compute gradients and use them in loss-minimizing parameter updates.
Read how Inside Deep Learning: Math, Algorithms, Models explains automatic differentiation →
Deep Learning with JAX
A technique for computing exact derivatives of code-expressed computations by systematically applying differentiation rules to the program structure.
- How it is used
- Supports gradient-based neural-network training and differentiable numerical programs within JAX's composable transformation system.
- How deep it goes
- Descends from practical grad() usage to forward and reverse modes, JVPs, VJPs, Jacobians, and Hessians.
- Examples
- Uses a scalar polynomial-rational function, a hand-derived derivative, JAX grad(), and neural-network loss gradients.
- What you can do after
- Apply it via JAX grad(), value_and_grad(), jacfwd(), jacrev(), hessian(), jvp(), and vjp() to obtain derivatives.
Read how Deep Learning with JAX explains automatic differentiation →
Deep Learning with R, Third Edition
Automatic differentiation computes exact derivatives of programs composed of differentiable operations by systematically applying the chain rule.
- How it is used
- Computes loss gradients for neural network weight updates without hand-writing derivative logic.
- How deep it goes
- Explains the chain rule and computation graphs, then implements a training step with GradientTape rather than backpropagation from scratch.
- Examples
- Uses a linear tensor expression and a per-batch neural network training step to demonstrate gradient computation.
- What you can do after
- Apply it via TensorFlow GradientTape to compute loss gradients for a Keras training step.
Read how Deep Learning with R, Third Edition explains automatic differentiation →
Build AI Drug Discovery Pipelines
Automatic differentiation is the systematic computation of exact derivatives of programs by applying the chain rule to their elementary operations.
- How it is used
- Trains neural networks by backpropagating loss gradients through computational graphs.
- How deep it goes
- Explains graph structure, tracked parameters, and backward gradient computation, but stops short of derivative internals.
- Examples
- Uses logistic regression with feature, weight, bias, sigmoid, binary-cross-entropy, and loss.backward() operations.
- What you can do after
- Apply it via PyTorch to compute parameter gradients from a model loss.
Read how Build AI Drug Discovery Pipelines explains automatic differentiation →
Deep Learning with PyTorch, Second Edition
A technique for automatically computing derivatives needed for training neural networks.
- How it is used
- Supports gradient-based parameter estimation by calculating how model loss changes with its parameters.
- How deep it goes
- Explains gradient tracking and computation graphs, then applies backward and parameter updates in a training loop.
- Examples
- Uses thermometer calibration with a linear model, mean squared loss, and a repeated parameter-update loop.
- What you can do after
- Apply it via PyTorch autograd to compute gradients and update model parameters.
Read how Deep Learning with PyTorch, Second Edition explains automatic differentiation →
Deep Learning with Python, Third Edition
A mechanism that programmatically computes exact derivatives of recorded operations by applying the chain rule to a computation graph.
- How it is used
- Provides gradients for differentiable functions in neural network computation and deep learning frameworks.
- How deep it goes
- Explains the chain rule and computation graphs as the basis for backpropagation, then identifies automatic differentiation as a framework feature.
- Examples
- Uses a chain of composed functions and a two-layer neural network loss calculation to illustrate differentiable operations.
- What you can do after
- Evaluate and understand how a computation graph supports automatic gradient calculation.
Read how Deep Learning with Python, Third Edition explains automatic differentiation →
Math and Architectures of Deep Learning
A method for mechanically computing derivatives of functions by tracing elementary operations.
- How it is used
- Scales gradient computation from simple formulas to nonlinear models with millions of parameters.
- How deep it goes
- Explains requires_grad, computation graphs, backward(), and optimizer updates, but does not derive Autograd internals.
- Examples
- Uses a noisy linear model and a demand-prediction neural network trained with mean squared error.
- What you can do after
- Apply it via PyTorch Autograd to compute gradients and train linear or neural-network models.
Read how Math and Architectures of Deep Learning explains automatic differentiation →
Grokking Bayes
Automatic differentiation is a technique that computationally tracks differentiable operations in a function to calculate its exact derivatives.
- How it is used
- Supports gradient-based optimization for complicated objectives in modern machine learning pipelines.
- How deep it goes
- Explains operation tracking, gradient flow, and sampling’s obstruction but stops short of implementation details.
- Examples
- Uses ELBO optimization and a water-pipe analogy to illustrate gradients passing through differentiable operations and stopping at sampling.
- What you can do after
- Apply it via gradient descent to optimize differentiable objectives without manually deriving their gradients.
Read how Grokking Bayes explains automatic differentiation →
Also covered in
- Machines that ThinkIn early access (MEAP)
- Evolutionary Deep Learning: Genetic algorithms and neural networks
What to know first
- gradientA gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- optimizationOptimization is the systematic selection or modification of values, configurations, or implementations to improve a defined objective, often subject to constraints.
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- chain ruleThe chain rule is a calculus rule for differentiating a composite function by combining the derivatives of its component functions.
Related concepts
- gradientproduces · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- JAXused by · JAX is a Python library for high-performance array computing, with composable transformations for automatic differentiation, compilation, vectorization, and parallelization.
- PyTorchused by · PyTorch is an open-source machine-learning framework that provides tensor computation, automatic differentiation, and tools for developing and training neural networks.
- computation graphuses · A computation graph is a directed graph whose nodes represent computational operations or values and whose edges represent dependencies between them.
- derivativeproduces · A derivative is a mathematical quantity that measures the instantaneous rate of change of a function with respect to an input variable.
- jax.gradimplemented by · jax.grad is a JAX transformation that returns a function computing gradients of a scalar-output function with respect to its inputs.
Questions about automatic differentiation
What is automatic differentiation?
Automatic differentiation is a computational technique that evaluates derivatives of a program by systematically applying the chain rule to its elementary operations.
Which Manning books explain automatic differentiation?
Inside Deep Learning: Math, Algorithms, Models, Deep Learning with JAX, Deep Learning with R, Third Edition, Build AI Drug Discovery Pipelines, Deep Learning with PyTorch, Second Edition, Deep Learning with Python, Third Edition, Math and Architectures of Deep Learning and Grokking Bayes.
What should I know before learning automatic differentiation?
Manning books that teach automatic differentiation usually introduce gradient, optimization, language model and chain rule first.
What is automatic differentiation also called?
Automatic differentiation is also called autograd, autodiff or autodifferentiation.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.







