A neuron combines its inputs and weights with a dot product, then applies an activation function before passing its output to the next layer. A nonlinear activation function lets the network represent more than a linear transformation. It can also keep neuron outputs within a predictable range. section 7.2 section 7.4
Sigmoid in Swift
func sigmoid(_ x: Double) -> Double { 1.0 / (1.0 + exp(-x)) } func derivativeSigmoid(_ x: Double) -> Double { sigmoid(x) * (1 - sigmoid(x)) } Sigmoid produces values between 0 and 1. Its computable derivative makes it usable during backpropagation. section 7.4
Keep both functions available to each neuron. During a forward pass, cache the dot product before applying the activation function. During backpropagation, evaluate the derivative at that cached value to calculate the neuron's delta. The input layer passes its inputs through; hidden and output layers apply their neurons' activation functions. section 7.5 section 7.5
// should only be called on an output layer
func calculateDeltasForOutputLayer(expected: [Double]) {
for n in 0..<neurons.count {
neurons[n].delta =
neurons[n].derivativeActivationFunction(neurons[n]
.outputCache) * (expected[n] - outputCache[n])
}
}
// should not be called on output layer
func calculateDeltasForHiddenLayer(nextLayer: Layer) {
for (index, neuron) in neurons.enumerated() {
let nextWeights = nextLayer.neurons.map { $0.weights[index] }
let nextDeltas = nextLayer.neurons.map { $0.delta }
let sumOfWeightsXDeltas = dotProduct(nextWeights, nextDeltas)
neuron.delta = neuron.derivativeActivationFunction(neuron
.outputCache) * sumOfWeightsXDeltas
}
}
}
Compare the output-layer and hidden-layer delta calculations: both call the activation function's derivative with the neuron's cached pre-activation value. section 7.5
To try another activation function:
- Supply both the replacement function and its derivative. section 7.5 section 7.9
- Compare classification accuracy and the amount of training needed. section 7.9
