A gradient tells you how a function changes as its variables change. In machine learning, it points to how model parameters should move to reduce a loss, making it central to optimization and learning. section 1.3 section 1.7
For one parameter, the gradient is the derivative. For parameters Θ, ∇Θ is a tensor containing one partial derivative for every parameter. Gradient descent updates parameters in the opposite direction: Θk+1 = Θk − η∇Θℓ, where η controls the step size. section 2.1.1
Computing a gradient with PyTorch
Set requires_grad=True on the variable, compute a scalar value with PyTorch operations, and call backward(). For x = -3.5 and f(x) = (x − 2)², value.backward() produces x.grad = -11. The recorded computation graph lets PyTorch perform the derivative calculation automatically. section 1.3.2
A neural-network training loop uses training data, a model and loss function, then gradients to update the model parameters. The loss measures performance; automatic differentiation supplies the parameter derivatives needed for the update. section 2.1
Gradients can become too small to produce meaningful parameter changes. Saturating activations such as tanh and sigmoid illustrate this vanishing-gradient problem: as their outputs stop changing, their derivatives approach zero and learning can stall. section 6.1.1
value.backward() print(x.grad) tensor([-11.])
Inspect the minimal PyTorch result showing a computed gradient after backward(). section 1.3.2
