A nonlinear mapping relates inputs and outputs in a way that repeated linear transformations alone cannot represent. Neural networks create this behavior by composing linear transformations with differentiable activation functions, allowing them to approximate complicated relationships without a handcrafted formula. section 6.1.6 section 6.1.5
Activation functions make the mapping useful for learning:
- Break repeated affine transformations out of a single linear form, enabling more complex functions. section 6.1.5
- Remain differentiable, or tolerate limited derivative discontinuities, so gradients can support training. section 6.1.5
- Provide sensitive input ranges where changes affect the output and saturated ranges where gradients become small. section 6.1.5
Composing neurons
A single tanh neuron produces a basic S-shaped response. Adding neurons creates bumps and curves; composing those outputs across two layers produces multiple positive and negative bumps that a single neuron cannot express. The same idea lets a network model tasks such as deciding whether an image contains a dog. section 6.1.6
Figure 6.6 Composing multiple linear units and tanh activation functions to produce nonlinear outputs
Trace how linear units and tanh activations combine into increasingly varied nonlinear outputs. section 6.1.6
