Automatic differentiation
Automatic differentiation is a computational technique that evaluates derivatives of a program by systematically applying the chain rule to its elementary operations.
Inside Deep Learning: Math, Algorithms, Models
A mechanism for computing derivatives of model computations automatically, used here to support training of networks with registered parameters.
- How it is used
- Supports loss minimization and neural-network training by supplying gradients for tensor computations and registered parameters.
- How deep it goes
- Explains derivative intuition, computation tracking, `.backward()`, manual updates, stopping criteria, and learning-rate behavior; it defers implementation details beyond PyTorch functions.
- Examples
- Uses f(x) = (x − 2)², x = −3.5, `.backward()`, x.grad = −11, iterative updates, and learning-rate convergence plots.
- What you can do after
- Apply it via PyTorch to compute gradients and build a basic gradient-descent update loop for scalar functions or neural-network losses.
Read how Inside Deep Learning: Math, Algorithms, Models explains automatic differentiation →
Deep Learning with JAX
A technique for computing exact derivatives of code-expressed computations by systematically applying differentiation rules to the program structure.
- How it is used
- Supports gradient-based neural-network training and differentiable numerical programs within JAX's composable transformation system.
- How deep it goes
- Descends from practical grad() usage to forward and reverse modes, JVPs, VJPs, Jacobians, and Hessians.
- Examples
- Uses a scalar polynomial-rational function, a hand-derived derivative, JAX grad(), and neural-network loss gradients.
- What you can do after
- Apply it via JAX grad(), value_and_grad(), jacfwd(), jacrev(), hessian(), jvp(), and vjp() to obtain derivatives.
Read how Deep Learning with JAX explains automatic differentiation →
Deep Learning with R, Third Edition
Automatic differentiation computes exact derivatives of programs composed of differentiable operations by systematically applying the chain rule.
- How it is used
- Supplies loss gradients needed for neural-network training without handwritten derivatives.
- How deep it goes
- Explains the chain-rule and computation-graph basis, then implements gradient retrieval without deriving softmax or cross-entropy gradients.
- Examples
- Uses an R image-classifier training step that records prediction and loss operations with `tf$GradientTape`.
- What you can do after
- Apply it via TensorFlow’s `tf$GradientTape` to obtain gradients of average loss with respect to model weights.
Read how Deep Learning with R, Third Edition explains automatic differentiation →
Deep Learning with PyTorch, Second Edition
A technique for automatically computing derivatives needed for training neural networks.
- How it is used
- Supplies parameter gradients for numerical optimization and model training without hand-deriving nested expressions.
- How deep it goes
- Works through gradient tracking, `backward()`, gradient clearing, and updates under `torch.no_grad()`; stops short of deriving individual operation derivatives.
- Examples
- Demonstrates thermometer calibration using a linear model, mean squared loss, parameter gradients, and a 5,000-epoch training loop.
- What you can do after
- Apply it via PyTorch autograd to calculate loss gradients and update a two-parameter calibration model without hand-derived derivatives.
Read how Deep Learning with PyTorch, Second Edition explains automatic differentiation →
Deep Learning with Python, Third Edition
A mechanism that programmatically computes exact derivatives of recorded operations by applying the chain rule to a computation graph.
- How it is used
- Computes gradients through a neural network’s loss so its weights can be trained.
- How deep it goes
- Explains the chain rule through function composition and computation graphs, then connects it to forward and backward passes.
- Examples
- Uses composed functions and a two-layer model’s loss expression to show the operations whose gradients must be calculated.
- What you can do after
- Evaluate/understand how frameworks calculate gradients through differentiable model operations instead of requiring handwritten gradient logic.
Read how Deep Learning with Python, Third Edition explains automatic differentiation →
Math and Architectures of Deep Learning
A method for mechanically computing derivatives of functions by tracing elementary operations.
- How it is used
- Scales gradient computation from simple formulas to nonlinear models with millions of parameters.
- How deep it goes
- Explains requires_grad, computation graphs, backward(), and optimizer updates, but does not derive Autograd internals.
- Examples
- Uses a noisy linear model and a demand-prediction neural network trained with mean squared error.
- What you can do after
- Apply it via PyTorch Autograd to compute gradients and train linear or neural-network models.
Read how Math and Architectures of Deep Learning explains automatic differentiation →
Related concepts
- PyTorchused by · PyTorch is an open-source machine-learning framework for Python that provides tensor computation, automatic differentiation, and tools for building and training neural networks.
- JAXused by · JAX is a Python library for high-performance array-based numerical computing with composable transformations including automatic differentiation, compilation, vectorization, and parallelization.
- derivativeproduces · A derivative is a mathematical quantity that measures the instantaneous rate of change of a function with respect to an input variable.
- computation graphuses · A computation graph is a directed graph whose nodes represent computational operations or values and whose edges represent data dependencies between them.
- gradientproduces · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- jax.gradimplemented by · jax.grad is a JAX transformation that creates a function computing derivatives, typically gradients of a scalar-output function with respect to specified inputs.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.