Gradient descent iteratively improves model parameters by moving them toward lower loss. Imagine descending a mountain: each step follows a direction expected to reach a lower point. In Gaussian-process training, the method adjusts hyperparameters such as the mean constant, length scale, and output scale to reduce the negative log likelihood. section 12.1.2 section 12.1.3
A typical training loop proceeds as follows:
- Define a likelihood-based loss, such as the negative exact marginal log likelihood. section 12.1.2
- Create an optimizer such as Adam and give it the parameters to update. section 12.1.2 section 3.3.2
- Run repeated iterations that clear gradients, compute the loss, backpropagate, and step the optimizer. section 12.1.2
- Switch the model and likelihood from training mode to evaluation mode before prediction. section 12.1.2
Optimizing a GP mean
A constant-mean GP can begin with a mean of zero while the data favor a mean of two. Adam repeatedly updates model.mean_module.constant to improve the likelihood; tracking the loss and constant shows whether they converge. section 3.3.2
optimizer = torch.optim.Adam(model.parameters(), lr=0.01) #1
mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, model) #2
model.train() #3
likelihood.train() #3
for i in tqdm(range(500)): #4
optimizer.zero_grad() #4
output = model(train_x) #4
loss = -mll(output, train_y) #4
loss.backward() #4
optimizer.step() #4
model.eval() #5
likelihood.eval() #5
The complete PyTorch and GPyTorch loop makes the update sequence directly reusable. section 12.1.2
Gradient descent depends on a reliable loss calculation. With large GP datasets, training can become slow and numerical instability can make the loss jagged, causing updates to produce a suboptimal model rather than a good minimum. section 12.1.3
