A hidden layer sits between a neural network’s input and output layers. In a feed-forward network, each neuron receives the preceding layer’s outputs and sends its own floating-point output to the next layer. These intermediate outputs are not the network’s final answer. A simple network can use one hidden layer; deeper networks use several. section 7.2
Figure 7.3. A simple neural network with one input layer of two neurons, one hidden layer of four neurons, and one output layer of three neurons. The number of neurons in each layer in this figure is arbitrary.
The diagram places four hidden neurons between two input neurons and three output neurons; those counts are illustrative, not required. section 7.2
In Swift, a network structure such as `[4, 6, 3]` specifies four input neurons, six hidden neurons, and three output neurons. A `Layer` processes incoming values through its neurons and caches the resulting outputs for the next layer. The input layer instead passes its inputs forward. The network repeats that handoff across its layers. section 7.5
Training also reaches the hidden layer. Unlike an output neuron, a hidden neuron has no expected output to compare directly with its own. Its delta is calculated from the next layer’s weights and deltas, together with its activation-function derivative. A subsequent weight-update step uses that delta, the previous layer’s cached outputs, and the learning rate. section 7.5 section 7.2
// should only be called on an output layer
func calculateDeltasForOutputLayer(expected: [Double]) {
for n in 0..<neurons.count {
neurons[n].delta =
neurons[n].derivativeActivationFunction(neurons[n]
.outputCache) * (expected[n] - outputCache[n])
}
}
// should not be called on output layer
func calculateDeltasForHiddenLayer(nextLayer: Layer) {
for (index, neuron) in neurons.enumerated() {
let nextWeights = nextLayer.neurons.map { $0.weights[index] }
let nextDeltas = nextLayer.neurons.map { $0.delta }
let sumOfWeightsXDeltas = dotProduct(nextWeights, nextDeltas)
neuron.delta = neuron.derivativeActivationFunction(neuron
.outputCache) * sumOfWeightsXDeltas
}
}
}
Compare the two Swift methods: output-layer deltas use expected values, while hidden-layer deltas use the next layer’s weights and deltas. section 7.5
Choosing a hidden-layer size
An iris classifier uses `[4, 6, 3]`: four measured inputs, six hidden neurons, and three species outputs. The six-neuron choice comes from trial and error rather than a formula. If accuracy is suboptimal, try changing the hidden-layer size or learning rate, then train and validate again. section 7.6
