An activation function transforms a neural network unit’s combined inputs before passing the result onward. Its nonlinearity allows successive layers to represent relationships that repeated linear transformations alone cannot capture. section 11.2.1 section 11.8
For inputs x₁, …, xₙ, weights w₁, …, wₙ, and bias b, a unit first computes z = w₁x₁ + … + wₙxₙ + b. Its activation is then a = ϕ(z), where ϕ is the activation function. The value z is the pre-activation; a becomes an input to the next layer. Weights determine the influence of each input, while the bias shifts the unit’s response. section 11.2.2 section 11.3.2 section 11.2.1
Each layer’s outputs form a representation: a numerical description shaped by its weights, biases, and activation functions. Geometrically, weighted combinations reorient and shift the data, while nonlinear activations introduce bends. Successive layers can build progressively more useful descriptions for prediction. section 11.2.1 section 11.2.2
Different activation functions transform the same pre-activation in different ways:
- Sigmoid maps inputs to values between 0 and 1; zero maps to 0.5. section 11.4.1
- Tanh maps inputs to values between −1 and 1 and is centered at zero. Positive and negative inputs produce outputs with the same respective signs. section 11.4.1
- ReLU is defined as max(0, z): it sets negative inputs to zero and passes positive inputs through unchanged. section 11.4.1
Comparing hidden-node activations
Suppose two hidden nodes receive the same input, x = 2.0. With weights and biases of 0.8 and −0.5 for the first node, and −0.4 and 0.9 for the second, their pre-activations are 1.1 and 0.1. Sigmoid produces approximately 0.75 and 0.52; tanh produces 0.80 and 0.10; ReLU produces 1.10 and 0.10. Thus the activation choice changes what the next layer receives even when the pre-activations are unchanged. section 11.3.2 section 11.4.1
A value between 0 and 1 is not automatically a probability. In the example, hidden-node sigmoid outputs are internal signals; the output-node sigmoid value represents the predicted probability of the positive class. Activation choice also changes gradients during learning, so it affects both representation and training. section 11.3.2 section 11.4.1



















