Backpropagation
Backpropagation is a procedure for computing derivatives of an output or loss with respect to inputs and parameters by applying the chain rule backward through a computational graph.
Math and Architectures of Deep Learning
The PyTorch method call that triggers backpropagation to compute gradients from the loss.
- How it is used
- Drives neural-network training by converting loss into parameter gradients before an optimizer updates weights and biases.
- How deep it goes
- Derives layerwise delta recurrences and parameter gradients, then maps the manual algorithm to PyTorch autograd.
- Examples
- Uses a single-neuron-per-layer MLP derivation and a PyTorch demand-prediction training loop with MSE and SGD.
- What you can do after
- Apply it via PyTorch by calling loss.backward() and optimizer.step() within a neural-network training loop.
Read how Math and Architectures of Deep Learning explains backpropagation →
Grokking Machine Learning
A neural-network training technique that computes loss derivatives with respect to model parameters by propagating error information backward through the network.
- How it is used
- Trains neural networks for classification and regression by iteratively reducing prediction error.
- How deep it goes
- Defines the algorithm and gives pseudocode, while deferring detailed one-hidden-layer derivative mathematics to an appendix.
- Examples
- Uses a circular two-feature classifier and MNIST handwritten-digit recognition with Keras training runs.
- What you can do after
- Apply it via Keras by defining a neural-network architecture, loss, optimizer, epochs, and batch size.
Read how Grokking Machine Learning explains backpropagation →
Deep Learning with Python, Third Edition
A PyTorch tensor method that triggers backpropagation from a loss tensor and fills gradients on participating tensors.
- How it is used
- Computes gradients for PyTorch training steps and gradient-based ConvNet visualization.
- How deep it goes
- Explains the chain rule and computation graphs, then works through `.backward()` in a training step without deriving every operation’s derivative.
- Examples
- Uses a squared scalar and a pure-PyTorch linear classifier to show gradient computation and parameter updates.
- What you can do after
- Apply it via PyTorch `.backward()` to read gradients, update trainable values, and inspect input-image sensitivity.
Read how Deep Learning with Python, Third Edition explains backpropagation →
Deep Learning with R, Third Edition
Backpropagation is an algorithm for computing gradients of a neural network's loss with respect to its parameters by applying the chain rule backward through the network.
- How it is used
- Computes loss gradients for weight updates during neural-network training.
- How deep it goes
- Works through the chain rule and computation graphs, then demonstrates a PyTorch training step without deriving every layer’s gradient manually.
- Examples
- Uses a two-layer network, a handwritten-digit classifier, and a PyTorch linear classifier’s training step.
- What you can do after
- Apply it via PyTorch `loss$backward()` to obtain gradients, update parameters, and clear gradients before the next step.
Read how Deep Learning with R, Third Edition explains backpropagation →
Deep Learning with PyTorch, Second Edition
A PyTorch autograd call that performs reverse-mode differentiation.
- How it is used
- Computes loss gradients that guide parameter updates during model training.
- How deep it goes
- Works through a PyTorch backward call and training loop, explaining computation-graph traversal and gradient accumulation without exhaustively covering the autograd API.
- Examples
- Uses a two-parameter linear model, a manual training loop, and separate training and validation losses.
- What you can do after
- Apply it via PyTorch autograd to compute gradients, clear accumulated values, and update parameters without backpropagating validation loss.
Read how Deep Learning with PyTorch, Second Edition explains backpropagation →
Natural Language Processing in Action, Second Edition
Backpropagation is an algorithm for computing gradients of a model's loss with respect to its parameters by propagating error signals backward through the computation.
- How it is used
- Uses error gradients to train layered neural networks and learned CNN kernels.
- How deep it goes
- Defines gradients and weight updates through practical consequences, but stops short of derivative-level implementation mechanics.
- Examples
- Illustrates single-neuron learning, layered predictions, and CNN kernels learning predictive text patterns.
- What you can do after
- Evaluate/understand how neural networks distribute error-based weight adjustments during optimization.
Read how Natural Language Processing in Action, Second Edition explains backpropagation →
Grokking AI Algorithms, Second Edition
Backpropagation is an algorithm for computing gradients of a model's loss with respect to its parameters by propagating error derivatives backward through the model.
- How it is used
- Uses prediction loss to find how neural-network weights should change during training.
- How deep it goes
- Works through chain-rule gradients and a Python training loop, while leaving language-model gradient computation as a placeholder.
- Examples
- Illustrates backward error assignment with a relay team, a numerical weight update, and a Python training loop.
- What you can do after
- Build it from scratch in a simple neural-network training loop; evaluate/understand its role in language and image models.
Read how Grokking AI Algorithms, Second Edition explains backpropagation →
Grokking AI Applications
Backpropagation is an optimization procedure that propagates prediction error through a neural network to adjust its parameters.
- How it is used
- Explains iterative neural-network training by adjusting parameters to reduce prediction errors.
- How deep it goes
- Defines the repeated prediction-and-adjustment loop but stops short of mathematical gradients or implementation details.
- Examples
- Uses student guesses, text completion, GPT-3’s 45 TB dataset, and GPUs as intuitive training examples.
- What you can do after
- Understand how repeated parameter adjustments reduce errors during neural-network training.
Read how Grokking AI Applications explains backpropagation →
Optimization Algorithms: AI techniques for design, planning, and control problems
Backpropagation is an algorithm that computes derivatives of a neural-network loss with respect to its parameters by propagating error information backward through the network.
- How it is used
- Explains how supervised neural networks use prediction error to adjust weights during iterative training.
- How deep it goes
- Provides a procedural overview from initialization through updates but does not derive chain-rule equations or implement gradient calculations.
- Examples
- Uses a two-layer penguin-species classifier with four features, ten hidden units, three classes, and an 83-parameter representation.
- What you can do after
- Evaluate/understand the role of backward derivatives in neural-network training and distinguish them from PSO parameter search.
Inside AI
Backpropagation is a neural-network training technique that computes weight adjustments by propagating output error backward through the network.
- How it is used
- Trains neural networks to improve predictions by adjusting weights after comparing a guess with a correct answer.
- How deep it goes
- Explains prediction, error comparison, and weight adjustment through an example, without specifying how updates are calculated.
- Examples
- Uses a pictured 7 misclassified as 1 and a three-layer network diagram to illustrate training.
- What you can do after
- Evaluate/understand how prediction errors help train a neural network and why deep networks still rely on weight adjustment.
Data Without Labels
Backpropagation is an algorithm that computes parameter gradients by propagating error signals backward through a neural network.
- How it is used
- Trains supervised neural networks by feeding prediction error backward to guide parameter adjustments.
- How deep it goes
- Works through layerwise gradients and the chain rule, with mathematical notation; specifies no software implementation.
- Examples
- Uses a backward-flow diagram and supervised dogs-versus-cats classification to illustrate the training context.
- What you can do after
- Evaluate/understand it as the gradient-calculation step that enables iterative weight updates and lower training loss.
Classic Computer Science Problems in Swift: Essential techniques for practicing programmers
- How it is used
- Trains feed-forward neural networks by propagating error corrections backward through their neurons.
- How deep it goes
- Explains output and hidden-layer deltas and weight updates, but uses calculus derivations rather than deriving them.
- Examples
- Uses output- and hidden-neuron delta diagrams and a sigmoid activation function with its derivative.
- What you can do after
- Apply the procedure to calculate neuron deltas and update network weights using a learning rate.
Learn Generative AI with PyTorch
A PyTorch call that computes gradients of the loss with respect to model parameters.
- How it is used
- Places gradient computation between loss evaluation and optimizer updates during neural-network training.
- How deep it goes
- Defines parameter gradients and shows the zero_grad(), backward(), and step() sequence, without deriving the underlying calculus.
- Examples
- Uses binary clothing classification and GAN training with discriminator and generator update functions.
- What you can do after
- Apply it via PyTorch by resetting gradients, calling backward(), and stepping an optimizer for each training batch.
Read how Learn Generative AI with PyTorch explains backpropagation →
Machine Learning Algorithms in Depth
Backpropagation computes gradients of an objective with respect to parameters by propagating derivatives backward through a computational composition.
- How it is used
- Supports parameter refinement in deep learning models as they minimize an objective function.
- How deep it goes
- Links composed layers to the chain rule and stochastic gradient descent, but does not supply a full backward-pass derivation or implementation.
- Examples
- Uses a sentiment-analysis network as an architecture example and poses a residual-connection backward pass as an exercise.
- What you can do after
- Practice/perform a backward-pass derivation for residual connections to explain their benefit.
Read how Machine Learning Algorithms in Depth explains backpropagation →
Evolutionary Deep Learning: Genetic algorithms and neural networks
Backpropagation is an algorithm that computes gradients of a neural-network objective with respect to parameters by propagating error signals backward through the network.
- How it is used
- Uses backpropagation as the conventional training baseline for comparing neuroevolutionary weight optimization.
- How deep it goes
- Explains backward loss propagation and contrasts optimization methods, but stops short of deriving gradients or implementing backpropagation internals.
- Examples
- Demonstrates the baseline with a NumPy MLP on classification, moons, circles, blobs, and Gaussian-quantile datasets.
- What you can do after
- Evaluate and understand when gradient-based backpropagation differs from evolutionary weight optimization.
LLMs in Production
Backpropagation is the procedure for computing parameter gradients by propagating error derivatives backward through a neural network.
- How it is used
- Trains language models by converting prediction error into parameter updates.
- How deep it goes
- Explains its role in minimizing loss but stops short of derivative calculations or implementation internals.
- Examples
- Uses next-word prediction, tokenized text, and a small from-scratch training example.
- What you can do after
- Practice a miniature from-scratch language-model training workflow using the provided code.
Sutskever's List
- How it is used
- Trains multilayer artificial neural networks by computing parameter gradients that guide updates toward lower prediction error.
- How deep it goes
- Explains gradient computation and its role in training, without deriving the chain rule or detailing implementation.
- What you can do after
- Understand how parameter gradients enable neural-network training to reduce prediction error.
Related concepts
- gradientproduces · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- chain ruleuses · The chain rule is a calculus rule for differentiating a composite function by multiplying derivatives along its composition.
- optimizationimplemented by · Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.
- objective functionuses · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.
- language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- forward passcontrasts with · A forward pass is a calculation that propagates inputs through an ordered computational or dependency structure to produce outputs or derived values.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.