Suppose a model’s loss is 0.5 with one coefficient set to 0.3. Raising it to 0.35 makes the loss 0.6; lowering it to 0.25 makes the loss 0.4. Testing both directions for every coefficient would take too many forward passes. A gradient gives the local change in loss for all the coefficients at once. section 2.4
A gradient generalizes a derivative to a function with tensor inputs. For a fixed sample and target, treat the loss as a function of the weight matrix W. Its gradient has the same shape as W: each entry indicates the direction and magnitude of the local loss change associated with changing that weight. section 2.4.2
y_pred = matmul(x, W) #1 loss_value = loss(y_pred, y_true) #2
This forward pass produces a prediction and a loss. Hold x and y_true fixed to see the loss as a function of W; the gradient describes how that loss changes with W. section 2.4.2
Two practical uses follow from that local signal:
- To train, compute the gradient on a batch and move parameters a little in the opposite direction. The learning rate controls the move; an optimizer determines exactly how to use the gradient. section 2.6
- To increase a filter’s response in an input image, use gradient ascent. Normalize the gradient tensor by its L2 norm to keep update magnitudes in a similar range. section 10.2
Differentiable tensor operations make gradients computable through a chain of operations using the chain rule, a process called backpropagation. When writing a low-level gradient loop, call model(x) rather than predict(): predict() returns outputs but cannot be backpropagated through. Gradient calculations use backend APIs such as TensorFlow GradientTape, PyTorch .backward(), or JAX jax.grad(). section 2.6 section 10.2
