A neuron scales an input by a weight, adds a bias, then applies an activation function: o = f(w × x + b). The activation is a fixed nonlinear operation, not another weight to learn. Without it, stacking linear transformations still produces a function of the same affine-linear form. With it, a network can produce different slopes at different input values and fit more complex relationships. section 6.1 section 6.1.5
Inside a network, activations help shape the function it learns. At the output, an activation can instead put values into a range suited to the task, such as a probability for binary classification. Activations also need to pass useful gradients backward during training. In a saturated range, changing the input barely changes the output, so the gradient can become close to zero. section 6.1.3 section 6.1.5 section 7.2.3
Common choices differ in shape and output range:
- Tanh is a smooth activation used between layers in the temperature-conversion network and after convolutions in an image classifier. section 6.1.4 section 6.3.2 section 8.2.4
- ReLU is a widely used general activation. LeakyReLU gives its negative-input side a small positive slope instead of zero. section 6.1.4
- Sigmoid maps an output into the 0–1 range when a single probability is wanted. section 6.1.4 section 7.2.3
Put an activation between linear layers
For a model with one input and one output, `nn.Sequential(nn.Linear(1, 8), nn.Tanh(), nn.Linear(8, 1))` applies Tanh to the eight hidden values before the output layer. The linear layers have trainable weights and biases; `nn.Tanh()` has no trainable parameters. section 6.3.2
Figure 6.5 A collection of common and not-so-common activation functions
Compare the shapes of Tanh, ReLU, Sigmoid, and other activations when choosing how a layer should respond to its inputs. section 6.1.4
