Backpropagation

As explained in Evolutionary Deep Learning: Genetic algorithms and neural networks

Backpropagation is an algorithm that computes gradients of a neural-network objective with respect to parameters by propagating error signals backward through the network.

Backpropagation trains neural networks by pushing calculated loss backward through the network. It is the conventional alternative to evolutionary parameter search. section 6.1 section 6.2

The treatment positions backpropagation as a baseline for comparison:

• Gradient descent and Adam optimize the network after loss is propagated backward. section 6.2
• Neuroevolution removes train and back_prop functions, replacing gradient-based updates with direct weight assignment from genetic-algorithm individuals. section 6.2
• Automatic differentiation and backpropagation suit continuous problems, while evolutionary optimization is presented for discontinuous problems. section 6.6

A simple MLP baseline

A NumPy multilayer perceptron trains on generated classification, moons, circles, blobs, or Gaussian-quantile datasets, then reports predictions and accuracy. The circles dataset provides a more difficult test than the simpler examples. section 6.1

Figure 6.1 Backpropagation vs. neuroevolution optimization

Compare the diagrammed flow of loss through an MLP with neuroevolution’s replacement of network weights. section 6.1

How Evolutionary Deep Learning: Genetic algorithms and neural networks covers it

How deep it goes

Explains backward loss propagation and contrasts optimization methods, but stops short of deriving gradients or implementing backpropagation internals.

Examples

Demonstrates the baseline with a NumPy MLP on classification, moons, circles, blobs, and Gaussian-quantile datasets.

What it assumes

Assumes a solid deep-learning background while reviewing the MLP structure and internal processes from a basic implementation.

What you can do after

Evaluate and understand when gradient-based backpropagation differs from evolutionary weight optimization.

How it connects

Connects backpropagation with neural networks, gradient descent, Adam, automatic differentiation, and neuroevolution.

liveBookmanning.com

Read Evolutionary Deep Learning: Genetic algorithms and neural networks on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Math and Architectures of Deep LearningThe PyTorch method call that triggers backpropagation to compute gradients from the loss.
• 

  liveBookmanning.com
  Grokking Machine LearningA neural-network training technique that computes loss derivatives with respect to model parameters by propagating error information backward through the network.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA PyTorch tensor method that triggers backpropagation from a loss tensor and fills gradients on participating tensors.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionBackpropagation is an algorithm for computing gradients of a neural network's loss with respect to its parameters by applying the chain rule backward through the network.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionA PyTorch autograd call that performs reverse-mode differentiation.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionBackpropagation is an algorithm for computing gradients of a model's loss with respect to its parameters by propagating error signals backward through the computation.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionBackpropagation is an algorithm for computing gradients of a model's loss with respect to its parameters by propagating error derivatives backward through the model.
• 

  liveBookmanning.com
  Grokking AI ApplicationsBackpropagation is an optimization procedure that propagates prediction error through a neural network to adjust its parameters.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsBackpropagation is an algorithm that computes derivatives of a neural-network loss with respect to its parameters by propagating error information backward through the network.
• 

  liveBookmanning.com
  Inside AIBackpropagation is a neural-network training technique that computes weight adjustments by propagating output error backward through the network.
• 

  liveBookmanning.com
  Data Without LabelsBackpropagation is an algorithm that computes parameter gradients by propagating error signals backward through a neural network.
• 

  liveBookmanning.com
  Classic Computer Science Problems in Swift: Essential techniques for practicing programmers
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchA PyTorch call that computes gradients of the loss with respect to model parameters.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthBackpropagation computes gradients of an objective with respect to parameters by propagating derivatives backward through a computational composition.
• 

  liveBookmanning.com
  LLMs in ProductionBackpropagation is the procedure for computing parameter gradients by propagating error derivatives backward through a neural network.
• 

  liveBookmanning.com
  Sutskever's List

Compare how 17 books explain backpropagation →

Related concepts

• gradientproduces · A gradient is the vector of partial derivatives of a scalar function with respect to its variables, indicating its local rate and direction of change.
• chain ruleuses · The chain rule is a calculus rule for differentiating a composite function by multiplying derivatives along its composition.
• optimizationimplemented by · Optimization is the systematic selection or adjustment of values, actions, or system designs to improve a specified objective, often subject to constraints.
• objective functionuses · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.
• language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• forward passcontrasts with · A forward pass is a calculation that propagates inputs through an ordered computational or dependency structure to produce outputs or derived values.

This explainer was generated by Manning's knowledge graph from the text of Evolutionary Deep Learning: Genetic algorithms and neural networks. The section links point to the parts of the book it draws on.