Stochastic gradient descent, commonly implemented as mini-batch gradient descent, updates model parameters using the loss and gradients estimated from small subsets of training data. It makes optimization practical when computing the full loss is expensive, as in Gaussian-process models whose exact training cost grows cubically with dataset size. section 12.0 section 12.2.2
Apply mini-batch gradient descent by repeating this training loop:
- Split the training set into small batches, such as ten batches of 100 points for a dataset of 1,000 points. section 12.2.2
- Compute the model output and loss for one batch rather than the entire dataset. section 12.2.2
- Backpropagate the batch loss, update the parameters, and repeat for every batch across many iterations. section 12.2.2 section 12.2.3
- Expect individual updates to follow imperfect directions; aggregated updates can still reduce the true loss effectively. section 12.2.2 section 12.5
for epoch in range(2): #1
running_loss = 0.0
for i, data in enumerate(trainloader, 0):
inputs, labels = data #2
optimizer.zero_grad() #3
outputs = net(inputs) #4
loss = criterion(outputs, labels) #4
loss.backward() #4
optimizer.step() #4
A compact PyTorch loop shows the essential sequence: clear gradients, compute outputs and loss, backpropagate, and step the optimizer. section 12.2.2
Training a variational Gaussian process
A variational Gaussian process uses mini-batch gradient descent to minimize an ELBO based on each batch. In GPyTorch, create a DataLoader, compute loss as -mll(output, y_batch), call loss.backward(), and update with optimizer.step(). The resulting predictions can roughly match those of an exact GP while training on medium-to-large datasets. section 12.2.3 section 12.5
Housing-price exercise
Compare exact and variational Gaussian processes on a housing-price dataset. Train the exact model and a variational model with 100 inducing points, use batches of 100 for mini-batch gradient descent, and compare training times. section 12.4
