A neural network layer is a stage of perceptrons that receives activations from the immediately preceding layer and produces a new set of activations. Connections run only between successive layers, making forward evaluation and training more manageable. Multiple layers can represent complicated functions with fewer perceptrons because each layer adds nonlinearity and expressive power. section 8.2 section 8.6
For layer l, the outputs of layer l−1 form the input activation vector a^(l−1). A linear layer combines these activations with a weight matrix W^(l) and biases, producing each neuron's weighted sum; applying an activation function such as sigmoid gives a^(l) = σ(W^(l)a^(l−1) + b^(l)). Matrix-vector notation expresses all neurons together instead of writing separate sums. section 8.3.1 section 8.6
A layered network therefore provides:
- A structured transformation from input space toward output space. section 1.8
- Additional nonlinear transformations that help model functions such as XOR and other complicated relationships. section 8.2
- Parameters—weights and biases—that training adjusts to reduce loss. section 8.6
From one layer to the next
If layer l−1 has m activations and layer l has n neurons, W^(l) is an n×m matrix. Each row collects the weights feeding one neuron; matrix multiplication computes all n weighted sums at once, after which sigmoid can be applied element by element. section 8.3.1
Figure 8.6 Linear layer outputting layer l from layer (l − 1). The weights belonging to row 1 of the weight matrix (coming from all the input neurons, layer (l − 1), which sum together to form output neuron 1) are shown in bold.
Use the figure to trace how one row of the weight matrix gathers activations from layer l−1 to produce a neuron in layer l. section 8.3.1
