An activation function transforms a neuron's combined input into its response. By introducing nonlinearity, it helps a neural network model patterns and boundaries that weighted sums alone cannot capture. section 3.1.1 section 3.6
A neuron scales its inputs by weights, adds the results and a bias, then applies an activation function. The resulting output can pass information to other parts of the network. The function shapes that response: it can pass through, reduce, or suppress a signal. section 3.1.1
Figure 3.5 A deep network without activation functions collapses into a single linear transformation. No matter how many layers are stacked, their combined effect is equivalent to a single layer, which would limit the model to learning only simple, linearly separable patterns.

Without nonlinear activation functions, stacking layers still produces only a single linear transformation. section 3.1.4
Common activation functions shape signals in different ways:
- Sigmoid maps inputs between zero and one. Its output was useful for early classification, but learning can be slow in deep networks. section 3.1.4
- Tanh centers its output around zero, supporting positive and negative signals. Its smoothness can also limit its usefulness in very deep models. section 3.1.4
- ReLU outputs zero for negative inputs and passes positive inputs through linearly. It helped training signals flow through many layers. section 3.1.4
- GELU scales inputs smoothly: small or negative signals can still influence the output, while stronger positive signals pass through more fully. It is common in modern deep networks, including Transformer-based language models. section 3.1.4
Figure 3.4 The ReLU activation function transforms a neuron’s input by suppressing negative values and passing positive ones through unchanged. This simple nonlinear behavior changes how inputs are separated, enabling the network to move beyond straight-line boundaries and capture more complex patterns.

ReLU illustrates how a simple nonlinear response can help a network move beyond straight-line boundaries. section 3.1.4



















