A neuron first multiplies its inputs by weights and adds the results. An activation function transforms that sum into the value the neuron passes onward. Think of it as the rule a neuron uses to turn a weighted signal into an output. Without a nonlinear activation function, a neural network can behave much like linear regression, limiting the patterns it can learn. section 9.2 section 9.10
In forward propagation, each hidden node applies an activation function to its weighted sum. The hidden-layer outputs are then weighted and summed for the output node, which applies an activation function again. In a simple NumPy implementation, these two steps take the form `sigmoid(np.dot(inputs, input_weights))` and `sigmoid(np.dot(hidden, hidden_weights))`. section 9.4 section 9.23
Choose the rule to suit what a layer needs to produce:
- A step function gives exactly 0 or 1, useful for a final binary decision but not for learning in a hidden layer. section 9.10
- Sigmoid gives values between 0 and 1 and supports small output changes as its input changes. Near its extremes, tiny derivatives can cause poor learning—the vanishing-gradient problem. section 9.10
- Hyperbolic tangent gives values between -1 and 1. Its steeper derivatives can allow faster learning, although it also has a vanishing-gradient problem at the extremes. section 9.10
- ReLU replaces negative values with 0 and leaves positive values unchanged. It provides a simple nonlinear step after weighted calculations. section 11.25
Activating a projected token
Suppose the first three weighted results are `[1.5921, 0.3436, -0.5260]`. Add the corresponding biases `[0.1, -0.05, 0.2]` to get `[1.6921, 0.2936, -0.3260]`. ReLU turns that into `[1.6921, 0.2936, 0]`: the negative result is removed, while the positive results pass through. section 11.25
Use the same activation function for nodes within one layer, but different layers can serve different purposes. For instance, a loan-decision network can use sigmoid in hidden layers and a step function at the output for a clear 0-or-1 decision. Activation choice also matters during training: backpropagation uses the activation function’s slope to calculate how errors relate to weights. section 9.16 section 9.26
Figure 9.29 Common used activation functions
Compare the shapes of common activation functions to see how differently they transform an input. section 9.10
