An activation function, also called a transfer function, determines a neuron's output after its inputs have been weighted and combined. That output can become input to the next layer. In a simple threshold perceptron, the neuron produces a nonzero output only when its weighted input exceeds a threshold. section 8.2.2 section 8.2.4
Nonlinearity matters: using only linear activation functions leaves the network's output linear, while a linear function has a constant derivative. Nonlinear functions therefore provide more scope for learning. section 8.2.4
Common choices serve different positions in a network:
- Sigmoid has an S-shaped curve and is used for an output between 0 and 1, often in the final layer for binary classification. section 8.2.4
- TANH is a smooth function generally used in hidden layers; centering values nearer zero can make training easier for the next layer. section 8.2.4
- ReLU uses F(x) = max(0, x). It is a favored hidden-layer choice because it is simple to use and less expensive to train. section 8.2.4
Face identification
Images of faces enter a network whose hidden layers process information using weights, biases, and activation functions. Feed-forward propagation carries the result toward a prediction, such as a probability score for a person's identity. section 8.3.2 section 8.2.1
Figure 8.5 A sigmoid function. Note the shape of the function and the min/max values.
The sigmoid plot makes its S-shaped curve and bounded output easier to inspect. section 8.2.4
