Backpropagation trains a feed-forward neural network by identifying errors after signals pass through it and sending corrective adjustments backward, especially to neurons most responsible for those errors. Neurons combine inputs and weights, then apply an activation function; that function needs a computable derivative for backpropagation. Sigmoid is one example, with outputs between 0 and 1. section 7.2 section 7.4
Training proceeds through related calculations:
- Calculate a delta for an output neuron. section 7.2
- Calculate deltas for neurons in hidden layers. section 7.2
- Update hidden- and output-layer weights using the deltas, prior weights and inputs, and a user-chosen learning rate. section 7.2
Figure 7.4. The mechanism by which an output neuron’s delta is calculated during the backpropagation phase of training
The diagram shows how an output neuron's delta is calculated. section 7.2
The diagram shows how a hidden-layer neuron's delta is calculated. section 7.2
Figure 7.6. The weights of every hidden layer and output layer neuron are updated using the deltas calculated in the previous steps, the prior weights, the prior inputs, and a user-determined learning rate.
The diagram shows weight updates based on deltas, prior values, and a learning rate. section 7.2
