A learning rate controls how far a neural network’s weights move during each training update. A neuron’s delta supplies the direction of the change; the learning rate scales its size. Finding a useful value for an unfamiliar problem takes trial and error. section 7.2
In a Swift network, the update follows three steps:
- Run the inputs forward and cache each layer’s outputs. section 7.5
- Compare the output with the expected result and calculate neuron deltas by backpropagation. This step does not change weights. section 7.5
- For each weight outside the input layer, add learningRate × previous-layer output × neuron delta to its existing value. section 7.5
/// backpropagate() doesn't actually change any weights
/// this function uses the deltas calculated in backpropagate()
/// to actually make changes to the weights
func updateWeights() {
for layer in layers.dropFirst() { // skip input layer
for neuron in layer.neurons {
for w in 0..<neuron.weights.count {
neuron.weights[w] = neuron.weights[w] +
(neuron.learningRate * (layer.previousLayer?
.outputCache[w])! * neuron.delta)
}
}
}
}
The Swift updateWeights() implementation shows exactly where each neuron’s learningRate scales a weight change. section 7.5
Tune against classification results
An iris classifier starts with Network(layerStructure: [4, 6, 3], learningRate: 0.3). It trains on 140 shuffled examples for 20 passes, then checks the remaining 10. A wine classifier uses [13, 7, 3] and a learning rate of 0.9. If accuracy is suboptimal, try changing the learning rate alongside the hidden-layer size and number of training passes. Random starting weights can make results vary between runs. section 7.6
