Backpropagation computes how a network’s loss changes with each weight by applying the chain rule backward through the network. Those gradients guide parameter updates through gradient descent. Popularized in the 1980s as a practical way to train multilayer artificial neural networks, it helped renew interest in the field. section 2.0
Backpropagation
Backpropagation is a procedure for computing derivatives of an output or loss with respect to inputs and parameters by applying the chain rule backward through a computational graph.
How Sutskever's List covers it
- How deep it goes
- Explains gradient computation and its role in training, without deriving the chain rule or detailing implementation.
- What it assumes
- Recaps weights, biases, prediction error, loss functions, and gradient descent while explaining the training context.
- What you can do after
- Understand how parameter gradients enable neural-network training to reduce prediction error.
- How it connects
- Connects artificial neural networks to loss functions and gradient descent.
Also explained in
-
Math and Architectures of Deep LearningThe PyTorch method call that triggers backpropagation to compute gradients from the loss.
-
Grokking Machine LearningA neural-network training technique that computes loss derivatives with respect to model parameters by propagating error information backward through the network.
-
Deep Learning with Python, Third EditionA PyTorch tensor method that triggers backpropagation from a loss tensor and fills gradients on participating tensors.
-
Deep Learning with R, Third EditionBackpropagation is an algorithm for computing gradients of a neural network's loss with respect to its parameters by applying the chain rule backward through the network.
-
Deep Learning with PyTorch, Second EditionA PyTorch autograd call that performs reverse-mode differentiation.
-
Natural Language Processing in Action, Second EditionBackpropagation is an algorithm for computing gradients of a model's loss with respect to its parameters by propagating error signals backward through the computation.
-
Grokking AI Algorithms, Second EditionBackpropagation is an algorithm for computing gradients of a model's loss with respect to its parameters by propagating error derivatives backward through the model.
-
Grokking AI ApplicationsBackpropagation is an optimization procedure that propagates prediction error through a neural network to adjust its parameters.
-
Optimization Algorithms: AI techniques for design, planning, and control problemsBackpropagation is an algorithm that computes derivatives of a neural-network loss with respect to its parameters by propagating error information backward through the network.
-
Inside AIBackpropagation is a neural-network training technique that computes weight adjustments by propagating output error backward through the network.
-
Data Without LabelsBackpropagation is an algorithm that computes parameter gradients by propagating error signals backward through a neural network.
-
Learn Generative AI with PyTorchA PyTorch call that computes gradients of the loss with respect to model parameters.
-
Machine Learning Algorithms in DepthBackpropagation computes gradients of an objective with respect to parameters by propagating derivatives backward through a computational composition.
-
Evolutionary Deep Learning: Genetic algorithms and neural networksBackpropagation is an algorithm that computes gradients of a neural-network objective with respect to parameters by propagating error signals backward through the network.
-
LLMs in ProductionBackpropagation is the procedure for computing parameter gradients by propagating error derivatives backward through a neural network.
Compare how 17 books explain backpropagation →
Related concepts
- gradientproduces · A gradient is a vector or tensor of partial derivatives that describes how a function changes with respect to its input variables or parameters.
- chain ruleuses · The chain rule is a calculus rule for differentiating a composite function by multiplying derivatives along its composition.
- optimizationimplemented by · Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.
- objective functionuses · An objective function is a function or criterion that assigns values to candidate solutions or parameter settings and is optimized by minimizing or maximizing those values.
- language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- forward passcontrasts with · A forward pass is a calculation that propagates inputs through an ordered computational or dependency structure to produce outputs or derived values.
This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.