liveBook · Manning

Backpropagation

Also called backward pass, backpropagate, loss.backward()

Backpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.

How 25 Manning books explain it

Math and Architectures of Deep Learning

The PyTorch method call that triggers backpropagation to compute gradients from the loss.

How it is used
Drives neural-network training by converting loss into parameter gradients before an optimizer updates weights and biases.
How deep it goes
Derives layerwise delta recurrences and parameter gradients, then maps the manual algorithm to PyTorch autograd.
Examples
Uses a single-neuron-per-layer MLP derivation and a PyTorch demand-prediction training loop with MSE and SGD.
What you can do after
Apply it via PyTorch by calling loss.backward() and optimizer.step() within a neural-network training loop.

Read how Math and Architectures of Deep Learning explains backpropagation →

Grokking Machine Learning

A neural-network training technique that computes loss derivatives with respect to model parameters by propagating error information backward through the network.

How it is used
Trains neural networks for classification and regression by iteratively reducing prediction error.
How deep it goes
Defines the algorithm and gives pseudocode, while deferring detailed one-hidden-layer derivative mathematics to an appendix.
Examples
Uses a circular two-feature classifier and MNIST handwritten-digit recognition with Keras training runs.
What you can do after
Apply it via Keras by defining a neural-network architecture, loss, optimizer, epochs, and batch size.

Read how Grokking Machine Learning explains backpropagation →

Deep Learning with Python, Third Edition

A PyTorch tensor method that triggers backpropagation from a loss tensor and fills gradients on participating tensors.

How it is used
Uses backward() to obtain gradients for training and for interpreting convolutional networks.
How deep it goes
Explains the chain rule and computation graphs, then demonstrates gradient tracking, accumulation, resetting, and parameter updates in PyTorch.
Examples
Demonstrates a squared scalar, a linear classifier training step, and input-image gradient ascent.
What you can do after
Apply it via PyTorch to compute gradients, update model parameters, and clear accumulated gradients.

Read how Deep Learning with Python, Third Edition explains backpropagation →

Deep Learning with R, Third Edition

Backpropagation is an algorithm for computing gradients of a neural network's loss with respect to its parameters by applying the chain rule backward through the network.

How it is used
Computes loss gradients during neural-network training so an optimizer can update the weights.
How deep it goes
Explains the chain rule and computation graphs, then demonstrates automatic differentiation and gradient handling in a training step.
Examples
Uses a two-layer handwritten-digit classifier and a linear classifier with mean squared error to illustrate gradient-based training.
What you can do after
Apply it via PyTorch’s backward method, manage gradient resets, and recognize its role in Keras fit.

Read how Deep Learning with R, Third Edition explains backpropagation →

Build AI Drug Discovery Pipelines

A backward pass is the reverse computation that propagates derivatives from a loss through a model to obtain parameter gradients.

How it is used
Guides parameter updates during neural network training by computing gradients of the loss.
How deep it goes
Explains the chain rule over a computational graph and walks through gradient calculation and parameter updates in a training loop.
Examples
Uses a logistic regression classifier with weights and bias, then a neural network training loop with batches and validation loss.
What you can do after
Apply it via PyTorch to compute parameter gradients and update a model during training.

Read how Build AI Drug Discovery Pipelines explains backpropagation →

Timeless Algorithms, The Foundational Ideas

A backward pass propagates learning signals from a model’s output back through its components to determine their contributions to the loss.

How it is used
Computes parameter sensitivities within the recurring neural-network training cycle, between loss measurement and optimization.
How deep it goes
Explains reverse application of the chain rule and works through output- and hidden-layer gradients in a numerical example.
Examples
Works through pedestrian detection using a 0.68 predicted probability, cross-entropy loss, and gradients before a parameter update.
What you can do after
Evaluate and understand how parameter gradients arise and why an optimizer, rather than backpropagation itself, changes weights and biases.

Read how Timeless Algorithms, The Foundational Ideas explains backpropagation →

Deep Learning with PyTorch, Second Edition

A PyTorch autograd call that performs reverse-mode differentiation.

How it is used
Uses the backward pass to calculate loss gradients for model training.
How deep it goes
Explains the chain rule, computation graph traversal, and gradient accumulation, then shows their place in training loops.
Examples
Demonstrates gradients for a linear model and a training loop with separate training and validation losses.
What you can do after
Apply it via PyTorch autograd to compute gradients, clear accumulated values, and update model parameters.

Read how Deep Learning with PyTorch, Second Edition explains backpropagation →

Natural Language Processing in Action, Second Edition

Backpropagation is an algorithm for computing gradients of a model's loss with respect to its parameters by propagating error signals backward through the computation.

How it is used
Uses error gradients to train layered neural networks and learned CNN kernels.
How deep it goes
Defines gradients and weight updates through practical consequences, but stops short of derivative-level implementation mechanics.
Examples
Illustrates single-neuron learning, layered predictions, and CNN kernels learning predictive text patterns.
What you can do after
Evaluate/understand how neural networks distribute error-based weight adjustments during optimization.

Read how Natural Language Processing in Action, Second Edition explains backpropagation →

Grokking AI Algorithms, Second Edition

Backpropagation is an algorithm for computing gradients of a model's loss with respect to its parameters by propagating error derivatives backward through the model.

How it is used
Uses backpropagation to train neural networks after forward propagation and loss calculation.
How deep it goes
Works through chain-rule gradients and a simple Python training loop, while sketching updates for a larger language model.
Examples
Uses a relay-team analogy, a single-weight update, and a neural-network training loop.
What you can do after
Build a simple neural-network training loop from scratch using forward propagation and backpropagation.

Read how Grokking AI Algorithms, Second Edition explains backpropagation →

Grokking AI Applications

Backpropagation is an optimization procedure that propagates prediction error through a neural network to adjust its parameters.

How it is used
Explains iterative neural-network training by adjusting parameters to reduce prediction errors.
How deep it goes
Defines the repeated prediction-and-adjustment loop but stops short of mathematical gradients or implementation details.
Examples
Uses student guesses, text completion, GPT-3’s 45 TB dataset, and GPUs as intuitive training examples.
What you can do after
Understand how repeated parameter adjustments reduce errors during neural-network training.

Read how Grokking AI Applications explains backpropagation →

Build a DeepSeek Model (From Scratch)

A backward pass is the computation that propagates gradients backward through a neural network to support parameter optimization.

How it is used
Explains how loss gradients propagate to earlier layers and support parameter updates during training.
How deep it goes
Works through linear-layer input and weight gradients, mixed-precision data flow, and distributed scheduling without deriving general backpropagation internals.
Examples
Uses a linear layer’s gradients, a PyTorch training step, and a multi-GPU pipeline drain as concrete examples.
What you can do after
Apply it via PyTorch using loss.backward(), optimizer.step(), and optimizer.zero_grad() in a training step.

Read how Build a DeepSeek Model (From Scratch) explains backpropagation →

Optimization Algorithms: AI techniques for design, planning, and control problems

Backpropagation is an algorithm that computes derivatives of a neural-network loss with respect to its parameters by propagating error information backward through the network.

How it is used
Explains how supervised neural networks use prediction error to adjust weights during iterative training.
How deep it goes
Provides a procedural overview from initialization through updates but does not derive chain-rule equations or implement gradient calculations.
Examples
Uses a two-layer penguin-species classifier with four features, ten hidden units, three classes, and an 83-parameter representation.
What you can do after
Evaluate/understand the role of backward derivatives in neural-network training and distinguish them from PSO parameter search.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains backpropagation →

Inside AI

Backpropagation is a neural-network training technique that computes weight adjustments by propagating output error backward through the network.

How it is used
Explains how neural networks use prediction errors to adjust weights during training.
How deep it goes
Describes error comparison and weight adjustment without detailing the calculation.
Examples
Illustrates training with a digit classifier that mistakes a 7 for a 1.
What you can do after
Evaluate and understand how prediction errors guide weight adjustments in neural-network training.

Read how Inside AI explains backpropagation →

Data Without Labels

Backpropagation is an algorithm that computes parameter gradients by propagating error signals backward through a neural network.

How it is used
Trains supervised neural networks by feeding prediction error backward to guide parameter adjustments.
How deep it goes
Works through layerwise gradients and the chain rule, with mathematical notation; specifies no software implementation.
Examples
Uses a backward-flow diagram and supervised dogs-versus-cats classification to illustrate the training context.
What you can do after
Evaluate/understand it as the gradient-calculation step that enables iterative weight updates and lower training loss.

Read how Data Without Labels explains backpropagation →

CUDA for LLMs

Backpropagation computes gradients by propagating derivatives backward through a neural network.

How it is used
Connects neural-network training with explicit gradient computation across framework and lower-level implementations.
How deep it goes
Derives scalar and batched gradients, then implements backward passes in NumPy, C, and CUDA.
Examples
Uses a scalar multiplication-and-square graph and a batched two-layer classifier to trace gradients.
What you can do after
Build a manual backward pass from scratch and check its training behavior against a PyTorch baseline.

Read how CUDA for LLMs explains backpropagation →

Classic Computer Science Problems in Swift: Essential techniques for practicing programmers

How it is used
Trains feed-forward neural networks by propagating error corrections backward through their neurons.
How deep it goes
Explains output and hidden-layer deltas and weight updates, but uses calculus derivations rather than deriving them.
Examples
Uses output- and hidden-neuron delta diagrams and a sigmoid activation function with its derivative.
What you can do after
Apply the procedure to calculate neuron deltas and update network weights using a learning rate.

Read how Classic Computer Science Problems in Swift: Essential techniques for practicing programmers explains backpropagation →

Learn Generative AI with PyTorch

A PyTorch call that computes gradients of the loss with respect to model parameters.

How it is used
Places gradient computation between loss evaluation and optimizer updates during neural-network training.
How deep it goes
Defines parameter gradients and shows the zero_grad(), backward(), and step() sequence, without deriving the underlying calculus.
Examples
Uses binary clothing classification and GAN training with discriminator and generator update functions.
What you can do after
Apply it via PyTorch by resetting gradients, calling backward(), and stepping an optimizer for each training batch.

Read how Learn Generative AI with PyTorch explains backpropagation →

Machine Learning Algorithms in Depth

Backpropagation computes gradients of an objective with respect to parameters by propagating derivatives backward through a computational composition.

How it is used
Supports parameter refinement in deep learning models as they minimize an objective function.
How deep it goes
Links composed layers to the chain rule and stochastic gradient descent, but does not supply a full backward-pass derivation or implementation.
Examples
Uses a sentiment-analysis network as an architecture example and poses a residual-connection backward pass as an exercise.
What you can do after
Practice/perform a backward-pass derivation for residual connections to explain their benefit.

Read how Machine Learning Algorithms in Depth explains backpropagation →

Fight Fraud with Machine Learning

Backpropagation is an algorithm that computes gradients of a loss with respect to neural-network parameters by propagating error derivatives backward through the network.

How it is used
Trains deep-learning classifiers and representation models for image, face, and voice fraud-detection tasks.
How deep it goes
Explains a complete gradient-descent workflow and chain-rule gradient propagation, but does not derive gradients for specific network layers.
Examples
Uses dog-versus-cat classification, same-versus-different face pairs, and a PyTorch fake-voice classifier.
What you can do after
Apply it via PyTorch to train parameters and distinguish training behavior from validation evaluation.

Read how Fight Fraud with Machine Learning explains backpropagation →

Applied Reinforcement Learning

Backpropagation computes gradients of a model loss with respect to parameters by applying the chain rule backward through a computational graph.

How it is used
Uses prediction loss to assign parameter gradients during neural-network learning in deep Q-learning.
How deep it goes
Explains chain-rule gradient flow through layers and parameter contributions, without implementation steps.
Examples
Illustrates gradients through layers of linear transformations and nonlinearities, using reinforcement-learning targets built from experience.
What you can do after
Understand how loss signals assign corrective gradients to weights and biases across network layers.

Read how Applied Reinforcement Learning explains backpropagation →

Machines that Think

A backward pass is the computation that propagates error information backward through a neural network to obtain parameter gradients.

How it is used
Explains how neural networks trace prediction error back to their weights during training.
How deep it goes
Explains weight sensitivities through derivatives and gradients, and identifies automatic differentiation as the practical computation method.
Examples
Illustrates sensitivity by asking how a slight change in a weight would affect the loss.
What you can do after
Evaluate or understand how gradients guide training and why training requires backward passes while inference typically does not.

Read how Machines that Think explains backpropagation →

Evolutionary Deep Learning: Genetic algorithms and neural networks

Backpropagation is an algorithm that computes gradients of a neural-network objective with respect to parameters by propagating error signals backward through the network.

How it is used
Uses backpropagation as the conventional training baseline for comparing neuroevolutionary weight optimization.
How deep it goes
Explains backward loss propagation and contrasts optimization methods, but stops short of deriving gradients or implementing backpropagation internals.
Examples
Demonstrates the baseline with a NumPy MLP on classification, moons, circles, blobs, and Gaussian-quantile datasets.
What you can do after
Evaluate and understand when gradient-based backpropagation differs from evolutionary weight optimization.

Read how Evolutionary Deep Learning: Genetic algorithms and neural networks explains backpropagation →

Grokking Machine Learning, Second Edition

A neural-network training algorithm that computes loss gradients with respect to model parameters by propagating error backward through the network.

How it is used
Trains neural networks by repeatedly computing loss gradients and updating weights and biases.
How deep it goes
Defines the training pseudocode and intuition, while deferring deeper-network mathematics to further resources.
Examples
Uses a single node, one data point, and changing edge weights to build a mental picture of training.
What you can do after
Apply it via Keras, TensorFlow, or PyTorch by defining a network architecture and error function.

Read how Grokking Machine Learning, Second Edition explains backpropagation →

LLMs in Production

Backpropagation is the procedure for computing parameter gradients by propagating error derivatives backward through a neural network.

How it is used
Trains language models by converting prediction error into parameter updates.
How deep it goes
Explains its role in minimizing loss but stops short of derivative calculations or implementation internals.
Examples
Uses next-word prediction, tokenized text, and a small from-scratch training example.
What you can do after
Practice a miniature from-scratch language-model training workflow using the provided code.

Read how LLMs in Production explains backpropagation →

Sutskever's List

How it is used
Trains multilayer artificial neural networks by computing parameter gradients that guide updates toward lower prediction error.
How deep it goes
Explains gradient computation and its role in training, without deriving the chain rule or detailing implementation.
What you can do after
Understand how parameter gradients enable neural-network training to reduce prediction error.

Read how Sutskever's List explains backpropagation →

Also covered in

What to know first

Related concepts

Questions about backpropagation

What is backpropagation?

Backpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.

Which Manning books explain backpropagation?

Math and Architectures of Deep Learning, Grokking Machine Learning, Deep Learning with Python, Third Edition, Deep Learning with R, Third Edition, Build AI Drug Discovery Pipelines, Timeless Algorithms, The Foundational Ideas, Deep Learning with PyTorch, Second Edition, Natural Language Processing in Action, Second Edition and 17 more.

What should I know before learning backpropagation?

Manning books that teach backpropagation usually introduce language model, objective function, gradient and parameter first.

What is backpropagation also called?

Backpropagation is also called backward pass, backpropagate or loss.backward().

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.