Expressive power is the variety and complexity of functions a neural network can represent. A network with greater expressive power can model more complicated input-output relationships; it is needed when a task demands a complex approximating function, such as recognizing dogs across large variations in pixels, foregrounds, and backgrounds. section 7.1 section 1.7 section 1.8
A neuron is a simple function, but weighted connections among many neurons can approximate complicated target functions from sample input-output pairs. Expressive power grows with the number of neurons and connections. Architecture fixes those structural choices; training then determines connection weights and biases. section 7.1 section 7.3 section 7.6
More expressive models are built by combining simple nonlinear units:
- Add nonlinearity and stack layers, so intermediate hidden outputs become inputs to later layers before a final output is formed. section 1.7
- Increase perceptrons or connections when the target relationship requires more representational capacity. section 7.1 section 7.6
- Use deeper convolutional networks for complex images: successive layers add learnable parameters and nonlinearities, with lower layers learning edges and higher layers learning more abstract shapes. section 11.2
Why XOR needs a multilayer network
A single perceptron can emulate AND, OR, and NOT, but not XOR. An MLP can represent XOR and other logical functions: treat each truth-table row as an AND of inputs, then combine those results with an OR. More generally, MLPs can approximate arbitrary functions; the geometric intuition is that shapes can be approximated by rectangles, and MLPs can construct such regions. section 7.6
More capacity is not automatically better. Too many perceptrons can cause overfitting, in which the network remembers training instances rather than developing general abstractions. Regularization and dropout counter this risk; dropout randomly turns off nodes during training, discouraging reliance on only a few nodes. section 9.3.6
Figure 11.4 A single 5 × 5 convolution layer vs. two 3 × 3 convolution layers
Compare one 5×5 convolution with two 3×3 convolutions: both give an output point the same 5×5 receptive field, while the deeper sequence adds nonlinearity and uses fewer parameters in the illustrated single-channel case. section 11.2
