Gradient descent minimizes an objective by repeatedly moving parameters in the direction opposite the objective's gradient. For model training, weights and biases descend the loss surface toward a low-loss point. section 8.4.2 section 9.2
If θ denotes the parameters and η the learning rate, the basic update is θ ← θ − η∇L(θ). The gradient must be recomputed after every update because the slope changes across the loss surface. section 8.4.2 section 8.4.3
The method combines geometric intuition with practical trade-offs:
- The negative gradient points toward the greatest local decrease in loss; η controls the step size. section 8.4.3
- A large learning rate can move quickly but may overshoot the minimum. section 8.4.3
- Nonconvex objectives can lead to local minima; restarting training from another random position may find a better minimum. section 8.4.4
- Momentum combines gradients from earlier iterations, strengthening consistent downhill movement and reducing noisy directional components. section 9.2.4
A PyTorch parameter update
A minimal optimization loop evaluates a loss, backpropagates its gradients, subtracts learning-rate-scaled gradients from μ and σ, and clears the stored gradients before the next iteration. section 6.8.2
MLE and MAP behavior
Gaussian maximum-likelihood estimation can converge when μ and σ start near their expected values but explode from distant initializations. Adding regularization for MAP prevents excessive parameter growth and enables convergence from farther away. section 6.8.2
Listing 6.6 Minimizing MLE loss via gradient descent
def minimize(X, mu, sigma, loss_fn, num_iters=100, lr = 0.001): #1
#2
for i in range(num_iters):
loss = loss_fn(X, mu, sigma) #3
loss.backward() #4
mu.data -= lr * mu.grad
sigma.data -= lr * sigma.grad #5
mu.grad.data.zero_()
sigma.grad.data.zero_() #6
mu = Variable(torch.Tensor([5]).type(dtype), requires_grad=True)
sigma = Variable(torch.Tensor([5]).type(dtype), requires_grad=True)
minimize(X, mu, sigma, neg_log_likelihood)
Inspect the complete PyTorch loop linking loss evaluation, backpropagation, parameter updates, and gradient reset. section 6.8.2
