An activation function derivative is a function that measures how an activation changes with respect to a neuron's input. A neuron stores it alongside its activation function so backpropagation can calculate error changes. section 7.5
In the neural-network implementation, the derivative:
- Arrives as a Swift function alongside the activation function and learning rate. section 7.5
- Evaluates the neuron's cached pre-activation output when calculating output-layer deltas. section 7.5
- Scales the weighted delta signal passed backward through hidden layers. section 7.5
Using the derivative during backpropagation
For an output neuron, the implementation multiplies derivativeActivationFunction(outputCache) by expected minus outputCache. Hidden neurons use the same derivative with the next layer's weighted deltas. section 7.5
/// An individual node in a layer
class Neuron {
var weights: [Double]
var activationFunction: (Double) -> Double
var derivativeActivationFunction: (Double) -> Double
var outputCache: Double = 0.0
var delta: Double = 0.0
var learningRate: Double
init(weights: [Double], activationFunction: @escaping (Double) ->
Double, derivativeActivationFunction: @escaping (Double) -> Double,
learningRate: Double) {
self.weights = weights
self.activationFunction = activationFunction
self.derivativeActivationFunction = derivativeActivationFunction
self.learningRate = learningRate
}
/// The output that will be going to the next layer
/// or the final output if this is an output layer
func output(inputs: [Double]) -> Double {
outputCache = dotProduct(inputs, weights)
return activationFunction(outputCache)
}
}
Inspect the Neuron definition to see the activation function, its derivative, cached input, and output method together. section 7.5
