Automatic relevance determination (ARD) equips a Gaussian-process kernel with one length scale per input dimension. This matters when an objective changes slowly along one axis but rapidly along another: a single shared length scale would misrepresent its variability. section 3.4.3 section 3.6
A length scale measures how quickly function values vary with an input. A large learned length scale indicates smoother, lower variability and therefore lower relevance; a small length scale indicates rapid variation and greater relevance. Gradient descent independently optimizes these scales with the other GP hyperparameters. section 3.4.3 section 3.6
Uneven variability in an objective
The two-dimensional Ackley function changes less across x than y, so ARD should learn a larger x length scale and a smaller y length scale. A trained model can expose these values, such as [0.7175, 0.4117]. section 3.4.3
In GPyTorch, set ard_num_dims to the number of inputs when constructing the covariance kernel. For example, RBFKernel(ard_num_dims=2) gives a two-dimensional GP separate length scales; omitting the argument creates a shared-scale comparison model. section 3.4.3
A practical ARD workflow is:
- Visualize or otherwise inspect whether the objective varies at different rates across dimensions. section 3.5
- Sample training points, fit the GP, and optimize its hyperparameters with gradient descent. section 3.4.3 section 3.5
- Inspect the learned scales and compare them with a model without ARD. section 3.5
Hyperparameter-tuning exercise
An SVM accuracy surface varies more quickly with the RBF parameter γ than with penalty parameter c. Training Matérn 5/2 GP models with and without ARD should reveal significantly different scales for these axes. section 3.5
