A derivative measures how sensitive a function is to a small change in its input. Think of standing on a hillside in the dark: you cannot see the valley, but you can feel the slope beneath your feet. The derivative gives you that slope at your current position. section 9.8
In neural-network training, the hillside is the cost function: it relates a weight to prediction error. Gradient descent uses its slope to choose a direction for adjusting the weight. A positive derivative means increasing that weight increases error, so decrease it. A negative derivative means increasing it decreases error, so increase it. A zero slope may mean you have reached a minimum. section 9.8 section 9.26
Figure 9.24 Derivatives’ slopes and direction of minimums
The slope at each position shows which direction leads toward a cost minimum. section 9.8
A weight deep inside a network affects cost through several steps: it changes a hidden-layer value, which changes a prediction, which changes the cost. Backpropagation works backward through those steps, multiplying their derivatives using the Chain Rule. The notation ∂J/∂W means the partial derivative of cost J with respect to one weight W; a layer’s gradient collects such derivatives for its weights. section 9.26
The slope of a sigmoid
An activation function also has a derivative. For sigmoid, the Python function `sigmoid_derivative(x)` returns `sigmoid(x) * (1 - sigmoid(x))`. During the backward pass, that slope helps determine how an output error should affect the weights. The network can then repeat forward propagation and backpropagation across training epochs. section 9.26 section 9.9
def sigmoid(x): return 1 / (1 + np.exp(-x)) def sigmoid_derivative(x): return sigmoid(x) * (1 - sigmoid(x)) #A
The short Python listing shows the sigmoid function beside its derivative. section 9.9
