A derivative measures the local slope of a function: how strongly its output changes when its input changes slightly. A positive derivative signals increase, a negative derivative signals decrease, and its magnitude measures the rate of change. section 2.4.1
From a small input change to optimization
Near a point p, a smooth function can be approximated by f(x + εx) = y + a · εx, where a is the derivative at p. To reduce f(x), adjust x slightly opposite the derivative’s direction. Familiar rules include d(cos x)/dx = −sin x and d(ax)/dx = a. section 2.4.1
Differentiation extends from scalar functions to tensor operations. The resulting gradient has the same shape as the input tensor and records how each input component changes the output, such as a loss value. section 2.4.2
For gradient-based learning, follow this practical chain:
- Use x and model weights W to compute predictions, then compare them with y_true through a loss function. section 2.4.2
- Interpret grad(loss_value, W0) as the direction and magnitude of loss change for each weight at the current W0. section 2.4.2
- Move weights in the direction that lowers the loss; this supplies the intuition behind gradient descent and neural-network learning. section 2.4.1 section 2.0
f(x + epsilon_x) = y + a * epsilon_x
Inspect the local linear approximation that connects a small input change with the derivative. section 2.4.1
y_pred = matmul(x, W) #1 loss_value = loss(y_pred, y_true) #2
Trace the concrete prediction-to-loss computation that turns model weights into an optimization target. section 2.4.2
