A Gaussian process becomes expensive as the training set grows. An inducing point is one of a small set of input locations used by a variational Gaussian process (VGP) to represent a much larger dataset. section 12.1.1 section 12.5
The points are not merely discarded observations. Their locations and associated variational function values are optimized so the resulting posterior GP captures the important trends of the full-data GP. section 12.2.1 section 12.2.3
A VGP uses inducing points through the following workflow:
- Initialize a variational distribution over the inducing points and a variational strategy that produces predictions from them. section 12.2.3
- Optimize the points, variational parameters, and ordinary GP parameters so the approximate posterior resembles the posterior conditioned on the entire dataset. section 12.2.1 section 12.2.3
- Use the evidence lower bound (ELBO) as a tractable training objective, often with mini-batches rather than evaluating every observation at once. section 12.5
if variational:
inducing_points =
model.variational_strategy.inducing_points.detach().clone()
with torch.no_grad():
inducing_mean = model(inducing_points).mean
plt.scatter(
inducing_points.squeeze(-1),
inducing_mean,
marker="D",
c="orange",
s=100,
label="inducing pts"
) #1
Visualize learned inducing locations and their predicted means alongside the observations to see which input regions summarize the dataset. section 12.1.1
Forrester-function toy dataset
With 1,000 randomly sampled inputs between −5 and 5, initialize a VGP from a smaller subset such as train_x[:50, :]. During training, learn the inducing locations and compare the VGP predictions with those of a regular GP. The learned points spread across locations that capture the objective's important trends, while the VGP roughly matches the full GP at lower training cost. section 12.1.1 section 12.2.3
Increasing the number of inducing points can improve representation but adds computation. The approximation therefore trades some fidelity for scalability, with mini-batch optimization improving practical loss computation on large datasets. section 12.4 section 12.5
