Backpropagation
Backpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.
Math and Architectures of Deep Learning
The PyTorch method call that triggers backpropagation to compute gradients from the loss.
- How it is used
- Drives neural-network training by converting loss into parameter gradients before an optimizer updates weights and biases.
- How deep it goes
- Derives layerwise delta recurrences and parameter gradients, then maps the manual algorithm to PyTorch autograd.
- Examples
- Uses a single-neuron-per-layer MLP derivation and a PyTorch demand-prediction training loop with MSE and SGD.
- What you can do after
- Apply it via PyTorch by calling loss.backward() and optimizer.step() within a neural-network training loop.
Read how Math and Architectures of Deep Learning explains backpropagation →
Grokking Machine Learning
A neural-network training technique that computes loss derivatives with respect to model parameters by propagating error information backward through the network.
- How it is used
- Trains neural networks for classification and regression by iteratively reducing prediction error.
- How deep it goes
- Defines the algorithm and gives pseudocode, while deferring detailed one-hidden-layer derivative mathematics to an appendix.
- Examples
- Uses a circular two-feature classifier and MNIST handwritten-digit recognition with Keras training runs.
- What you can do after
- Apply it via Keras by defining a neural-network architecture, loss, optimizer, epochs, and batch size.
Read how Grokking Machine Learning explains backpropagation →
Deep Learning with Python, Third Edition
A PyTorch tensor method that triggers backpropagation from a loss tensor and fills gradients on participating tensors.
- How it is used
- Uses backward() to obtain gradients for training and for interpreting convolutional networks.
- How deep it goes
- Explains the chain rule and computation graphs, then demonstrates gradient tracking, accumulation, resetting, and parameter updates in PyTorch.
- Examples
- Demonstrates a squared scalar, a linear classifier training step, and input-image gradient ascent.
- What you can do after
- Apply it via PyTorch to compute gradients, update model parameters, and clear accumulated gradients.
Read how Deep Learning with Python, Third Edition explains backpropagation →
Deep Learning with R, Third Edition
Backpropagation is an algorithm for computing gradients of a neural network's loss with respect to its parameters by applying the chain rule backward through the network.
- How it is used
- Computes loss gradients during neural-network training so an optimizer can update the weights.
- How deep it goes
- Explains the chain rule and computation graphs, then demonstrates automatic differentiation and gradient handling in a training step.
- Examples
- Uses a two-layer handwritten-digit classifier and a linear classifier with mean squared error to illustrate gradient-based training.
- What you can do after
- Apply it via PyTorch’s backward method, manage gradient resets, and recognize its role in Keras fit.
Read how Deep Learning with R, Third Edition explains backpropagation →
Build AI Drug Discovery Pipelines
A backward pass is the reverse computation that propagates derivatives from a loss through a model to obtain parameter gradients.
- How it is used
- Guides parameter updates during neural network training by computing gradients of the loss.
- How deep it goes
- Explains the chain rule over a computational graph and walks through gradient calculation and parameter updates in a training loop.
- Examples
- Uses a logistic regression classifier with weights and bias, then a neural network training loop with batches and validation loss.
- What you can do after
- Apply it via PyTorch to compute parameter gradients and update a model during training.
Read how Build AI Drug Discovery Pipelines explains backpropagation →
Timeless Algorithms, The Foundational Ideas
A backward pass propagates learning signals from a model’s output back through its components to determine their contributions to the loss.
- How it is used
- Computes parameter sensitivities within the recurring neural-network training cycle, between loss measurement and optimization.
- How deep it goes
- Explains reverse application of the chain rule and works through output- and hidden-layer gradients in a numerical example.
- Examples
- Works through pedestrian detection using a 0.68 predicted probability, cross-entropy loss, and gradients before a parameter update.
- What you can do after
- Evaluate and understand how parameter gradients arise and why an optimizer, rather than backpropagation itself, changes weights and biases.
Read how Timeless Algorithms, The Foundational Ideas explains backpropagation →
Deep Learning with PyTorch, Second Edition
A PyTorch autograd call that performs reverse-mode differentiation.
- How it is used
- Uses the backward pass to calculate loss gradients for model training.
- How deep it goes
- Explains the chain rule, computation graph traversal, and gradient accumulation, then shows their place in training loops.
- Examples
- Demonstrates gradients for a linear model and a training loop with separate training and validation losses.
- What you can do after
- Apply it via PyTorch autograd to compute gradients, clear accumulated values, and update model parameters.
Read how Deep Learning with PyTorch, Second Edition explains backpropagation →
Natural Language Processing in Action, Second Edition
Backpropagation is an algorithm for computing gradients of a model's loss with respect to its parameters by propagating error signals backward through the computation.
- How it is used
- Uses error gradients to train layered neural networks and learned CNN kernels.
- How deep it goes
- Defines gradients and weight updates through practical consequences, but stops short of derivative-level implementation mechanics.
- Examples
- Illustrates single-neuron learning, layered predictions, and CNN kernels learning predictive text patterns.
- What you can do after
- Evaluate/understand how neural networks distribute error-based weight adjustments during optimization.
Read how Natural Language Processing in Action, Second Edition explains backpropagation →
Grokking AI Algorithms, Second Edition
Backpropagation is an algorithm for computing gradients of a model's loss with respect to its parameters by propagating error derivatives backward through the model.
- How it is used
- Uses backpropagation to train neural networks after forward propagation and loss calculation.
- How deep it goes
- Works through chain-rule gradients and a simple Python training loop, while sketching updates for a larger language model.
- Examples
- Uses a relay-team analogy, a single-weight update, and a neural-network training loop.
- What you can do after
- Build a simple neural-network training loop from scratch using forward propagation and backpropagation.
Read how Grokking AI Algorithms, Second Edition explains backpropagation →
Grokking AI Applications
Backpropagation is an optimization procedure that propagates prediction error through a neural network to adjust its parameters.
- How it is used
- Explains iterative neural-network training by adjusting parameters to reduce prediction errors.
- How deep it goes
- Defines the repeated prediction-and-adjustment loop but stops short of mathematical gradients or implementation details.
- Examples
- Uses student guesses, text completion, GPT-3’s 45 TB dataset, and GPUs as intuitive training examples.
- What you can do after
- Understand how repeated parameter adjustments reduce errors during neural-network training.
Read how Grokking AI Applications explains backpropagation →
Build a DeepSeek Model (From Scratch)
A backward pass is the computation that propagates gradients backward through a neural network to support parameter optimization.
- How it is used
- Explains how loss gradients propagate to earlier layers and support parameter updates during training.
- How deep it goes
- Works through linear-layer input and weight gradients, mixed-precision data flow, and distributed scheduling without deriving general backpropagation internals.
- Examples
- Uses a linear layer’s gradients, a PyTorch training step, and a multi-GPU pipeline drain as concrete examples.
- What you can do after
- Apply it via PyTorch using loss.backward(), optimizer.step(), and optimizer.zero_grad() in a training step.
Read how Build a DeepSeek Model (From Scratch) explains backpropagation →
Optimization Algorithms: AI techniques for design, planning, and control problems
Backpropagation is an algorithm that computes derivatives of a neural-network loss with respect to its parameters by propagating error information backward through the network.
- How it is used
- Explains how supervised neural networks use prediction error to adjust weights during iterative training.
- How deep it goes
- Provides a procedural overview from initialization through updates but does not derive chain-rule equations or implement gradient calculations.
- Examples
- Uses a two-layer penguin-species classifier with four features, ten hidden units, three classes, and an 83-parameter representation.
- What you can do after
- Evaluate/understand the role of backward derivatives in neural-network training and distinguish them from PSO parameter search.
Inside AI
Backpropagation is a neural-network training technique that computes weight adjustments by propagating output error backward through the network.
- How it is used
- Explains how neural networks use prediction errors to adjust weights during training.
- How deep it goes
- Describes error comparison and weight adjustment without detailing the calculation.
- Examples
- Illustrates training with a digit classifier that mistakes a 7 for a 1.
- What you can do after
- Evaluate and understand how prediction errors guide weight adjustments in neural-network training.
Data Without Labels
Backpropagation is an algorithm that computes parameter gradients by propagating error signals backward through a neural network.
- How it is used
- Trains supervised neural networks by feeding prediction error backward to guide parameter adjustments.
- How deep it goes
- Works through layerwise gradients and the chain rule, with mathematical notation; specifies no software implementation.
- Examples
- Uses a backward-flow diagram and supervised dogs-versus-cats classification to illustrate the training context.
- What you can do after
- Evaluate/understand it as the gradient-calculation step that enables iterative weight updates and lower training loss.
CUDA for LLMs
Backpropagation computes gradients by propagating derivatives backward through a neural network.
- How it is used
- Connects neural-network training with explicit gradient computation across framework and lower-level implementations.
- How deep it goes
- Derives scalar and batched gradients, then implements backward passes in NumPy, C, and CUDA.
- Examples
- Uses a scalar multiplication-and-square graph and a batched two-layer classifier to trace gradients.
- What you can do after
- Build a manual backward pass from scratch and check its training behavior against a PyTorch baseline.
Classic Computer Science Problems in Swift: Essential techniques for practicing programmers
- How it is used
- Trains feed-forward neural networks by propagating error corrections backward through their neurons.
- How deep it goes
- Explains output and hidden-layer deltas and weight updates, but uses calculus derivations rather than deriving them.
- Examples
- Uses output- and hidden-neuron delta diagrams and a sigmoid activation function with its derivative.
- What you can do after
- Apply the procedure to calculate neuron deltas and update network weights using a learning rate.
Learn Generative AI with PyTorch
A PyTorch call that computes gradients of the loss with respect to model parameters.
- How it is used
- Places gradient computation between loss evaluation and optimizer updates during neural-network training.
- How deep it goes
- Defines parameter gradients and shows the zero_grad(), backward(), and step() sequence, without deriving the underlying calculus.
- Examples
- Uses binary clothing classification and GAN training with discriminator and generator update functions.
- What you can do after
- Apply it via PyTorch by resetting gradients, calling backward(), and stepping an optimizer for each training batch.
Read how Learn Generative AI with PyTorch explains backpropagation →
Machine Learning Algorithms in Depth
Backpropagation computes gradients of an objective with respect to parameters by propagating derivatives backward through a computational composition.
- How it is used
- Supports parameter refinement in deep learning models as they minimize an objective function.
- How deep it goes
- Links composed layers to the chain rule and stochastic gradient descent, but does not supply a full backward-pass derivation or implementation.
- Examples
- Uses a sentiment-analysis network as an architecture example and poses a residual-connection backward pass as an exercise.
- What you can do after
- Practice/perform a backward-pass derivation for residual connections to explain their benefit.
Read how Machine Learning Algorithms in Depth explains backpropagation →
Fight Fraud with Machine Learning
Backpropagation is an algorithm that computes gradients of a loss with respect to neural-network parameters by propagating error derivatives backward through the network.
- How it is used
- Trains deep-learning classifiers and representation models for image, face, and voice fraud-detection tasks.
- How deep it goes
- Explains a complete gradient-descent workflow and chain-rule gradient propagation, but does not derive gradients for specific network layers.
- Examples
- Uses dog-versus-cat classification, same-versus-different face pairs, and a PyTorch fake-voice classifier.
- What you can do after
- Apply it via PyTorch to train parameters and distinguish training behavior from validation evaluation.
Read how Fight Fraud with Machine Learning explains backpropagation →
Applied Reinforcement Learning
Backpropagation computes gradients of a model loss with respect to parameters by applying the chain rule backward through a computational graph.
- How it is used
- Uses prediction loss to assign parameter gradients during neural-network learning in deep Q-learning.
- How deep it goes
- Explains chain-rule gradient flow through layers and parameter contributions, without implementation steps.
- Examples
- Illustrates gradients through layers of linear transformations and nonlinearities, using reinforcement-learning targets built from experience.
- What you can do after
- Understand how loss signals assign corrective gradients to weights and biases across network layers.
Read how Applied Reinforcement Learning explains backpropagation →
Machines that Think
A backward pass is the computation that propagates error information backward through a neural network to obtain parameter gradients.
- How it is used
- Explains how neural networks trace prediction error back to their weights during training.
- How deep it goes
- Explains weight sensitivities through derivatives and gradients, and identifies automatic differentiation as the practical computation method.
- Examples
- Illustrates sensitivity by asking how a slight change in a weight would affect the loss.
- What you can do after
- Evaluate or understand how gradients guide training and why training requires backward passes while inference typically does not.
Evolutionary Deep Learning: Genetic algorithms and neural networks
Backpropagation is an algorithm that computes gradients of a neural-network objective with respect to parameters by propagating error signals backward through the network.
- How it is used
- Uses backpropagation as the conventional training baseline for comparing neuroevolutionary weight optimization.
- How deep it goes
- Explains backward loss propagation and contrasts optimization methods, but stops short of deriving gradients or implementing backpropagation internals.
- Examples
- Demonstrates the baseline with a NumPy MLP on classification, moons, circles, blobs, and Gaussian-quantile datasets.
- What you can do after
- Evaluate and understand when gradient-based backpropagation differs from evolutionary weight optimization.
Grokking Machine Learning, Second Edition
A neural-network training algorithm that computes loss gradients with respect to model parameters by propagating error backward through the network.
- How it is used
- Trains neural networks by repeatedly computing loss gradients and updating weights and biases.
- How deep it goes
- Defines the training pseudocode and intuition, while deferring deeper-network mathematics to further resources.
- Examples
- Uses a single node, one data point, and changing edge weights to build a mental picture of training.
- What you can do after
- Apply it via Keras, TensorFlow, or PyTorch by defining a network architecture and error function.
Read how Grokking Machine Learning, Second Edition explains backpropagation →
LLMs in Production
Backpropagation is the procedure for computing parameter gradients by propagating error derivatives backward through a neural network.
- How it is used
- Trains language models by converting prediction error into parameter updates.
- How deep it goes
- Explains its role in minimizing loss but stops short of derivative calculations or implementation internals.
- Examples
- Uses next-word prediction, tokenized text, and a small from-scratch training example.
- What you can do after
- Practice a miniature from-scratch language-model training workflow using the provided code.
Sutskever's List
- How it is used
- Trains multilayer artificial neural networks by computing parameter gradients that guide updates toward lower prediction error.
- How deep it goes
- Explains gradient computation and its role in training, without deriving the chain rule or detailing implementation.
- What you can do after
- Understand how parameter gradients enable neural-network training to reduce prediction error.
Also covered in
- Deep Learning with JAX
- Demand Forecasting Best Practices
- Inside Deep Learning: Math, Algorithms, Models
- AI Agents in Action, Second Edition
- Designing Deep Learning Systems: A guide for software engineers
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- objective functionAn objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
- gradientA gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- parameterA parameter is a value or named input that specifies or influences the behavior, computation, or output of a function, model, or system.
Related concepts
- gradientproduces · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
- chain ruleuses · The chain rule is a calculus rule for differentiating a composite function by combining the derivatives of its component functions.
- optimizationimplemented by · Optimization is the systematic selection or modification of values, configurations, or implementations to improve a defined objective, often subject to constraints.
- objective functionuses · An objective function is a mathematical function that assigns values to candidate solutions or parameter settings, which an optimization procedure seeks to minimize or maximize.
Questions about backpropagation
What is backpropagation?
Backpropagation is reverse-mode differentiation that propagates derivatives backward through a computational graph to compute gradients of an output or loss with respect to inputs and parameters.
Which Manning books explain backpropagation?
Math and Architectures of Deep Learning, Grokking Machine Learning, Deep Learning with Python, Third Edition, Deep Learning with R, Third Edition, Build AI Drug Discovery Pipelines, Timeless Algorithms, The Foundational Ideas, Deep Learning with PyTorch, Second Edition, Natural Language Processing in Action, Second Edition and 17 more.
What should I know before learning backpropagation?
Manning books that teach backpropagation usually introduce language model, objective function, gradient and parameter first.
What is backpropagation also called?
Backpropagation is also called backward pass, backpropagate or loss.backward().
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
























