A delta is a neuron's error signal during backpropagation. It combines an error term with the derivative of the neuron's activation function, evaluated at the cached value from before activation. Deltas are calculated before weights change. section 7.2 section 7.5
The error term depends on where the neuron sits:
- Output layer: multiply the activation derivative by expected output minus actual output. section 7.5
- Hidden layer: take the dot product of the next layer's relevant weights and deltas, then multiply by the activation derivative. section 7.5
// should only be called on an output layer
func calculateDeltasForOutputLayer(expected: [Double]) {
for n in 0..<neurons.count {
neurons[n].delta =
neurons[n].derivativeActivationFunction(neurons[n]
.outputCache) * (expected[n] - outputCache[n])
}
}
// should not be called on output layer
func calculateDeltasForHiddenLayer(nextLayer: Layer) {
for (index, neuron) in neurons.enumerated() {
let nextWeights = nextLayer.neurons.map { $0.weights[index] }
let nextDeltas = nextLayer.neurons.map { $0.delta }
let sumOfWeightsXDeltas = dotProduct(nextWeights, nextDeltas)
neuron.delta = neuron.derivativeActivationFunction(neuron
.outputCache) * sumOfWeightsXDeltas
}
}
}
The Swift layer methods show both calculations. Each neuron stores its resulting delta; the hidden-layer method reads deltas from the next layer. section 7.5
A delta does not change a weight by itself. For each weight, the update adds the learning rate multiplied by the previous layer's cached output and the neuron's delta. Calculate the needed deltas before using them to update weights. The partial-derivative derivation is left aside. section 7.2 section 7.5
/// backpropagate() doesn't actually change any weights
/// this function uses the deltas calculated in backpropagate()
/// to actually make changes to the weights
func updateWeights() {
for layer in layers.dropFirst() { // skip input layer
for neuron in layer.neurons {
for w in 0..<neuron.weights.count {
neuron.weights[w] = neuron.weights[w] +
(neuron.learningRate * (layer.previousLayer?
.outputCache[w])! * neuron.delta)
}
}
}
}
The weight-update code makes the distinction concrete: backpropagation stores deltas, while this separate method applies them to weights. section 7.5
