A nonlinear mapping produces outputs that are not merely linear transformations of inputs. In deep neural networks, activation functions supply this nonlinearity between weighted layers, allowing the model to learn complex patterns and decision boundaries. section 4.1.1
Nonlinearity matters because it:
- Prevents stacked neural-network layers from collapsing into one equivalent matrix multiplication. section 4.1.1
- Increases expressive power by enabling approximation of arbitrary functions and more sophisticated decision boundaries. section 4.1.1
- Appears through activation functions including Sigmoid, TanH, and ReLU, each offering different computational and gradient behavior. section 4.1.1
Figure 4.5 Some examples of activation functions commonly used in deep neural networks – Sigmoid, TanH, and ReLU.
Compare the shapes and ranges of Sigmoid, TanH, and ReLU, the activation functions used to introduce nonlinearity. section 4.1.1










