Training a neural network requires the gradient of its loss with respect to its weights. That loss may pass through many tensor operations, so writing a separate derivative calculation for every model is impractical. Automatic differentiation makes the calculation programmable: it represents the operations as a computation graph and applies the chain rule to obtain gradients. section 2.4.4 section 3.1
Start with two functions
Suppose x1 = g(x) and y = f(x1). The derivative of y with respect to x is grad(y, x1) × grad(x1, x). Add more intermediate functions, and the result becomes a longer chain of local derivatives. This is the calculation backpropagation applies to a neural network. section 2.4.4
def fghj(x): x1 = j(x) x2 = h(x1) x3 = g(x2) y = f(x3) return y grad(y, x) == grad(y, x3) * grad(x3, x2) * grad(x2, x1) * grad(x1, x)
The function composition and its gradient expression show how each intermediate result contributes one factor to the chain. section 2.4.4
A computation graph records operations as a directed acyclic graph rather than leaving them only as lines of code. A model’s forward computation can include matrix multiplication, addition, ReLU, softmax, and a loss function. The graph provides an explicit structure for calculating gradients through that sequence. section 2.4.4
loss_value = loss( y_true, softmax(matmul(relu(matmul(inputs, W1) + b1), W2) + b2), )
This nested loss expression shows why automatic differentiation matters in practice: the gradient must pass through several composed tensor operations. section 2.4.4
In practice, use a deep learning framework rather than implementing gradient logic from scratch. JAX, TensorFlow, PyTorch, and Keras provide automatic differentiation alongside tensor computation and distributed computation. The key condition remains that the function being differentiated is composed of differentiable operations. section 3.1 section 2.4.4
