A length scale controls how rapidly a Gaussian-process function changes as inputs become separated. A short length scale permits greater variation over a fixed input change; a long length scale produces smoother predictions and typically less uncertainty. section 3.4.1 section 3.6
The parameter belongs to the covariance function and is a GP hyperparameter. In GPyTorch, an RBFKernel stores it at model.covar_module.base_kernel.lengthscale, while ScaleKernel supplies the separate output scale that controls prediction range. section 3.4.1
class ScaleGPModel(gpytorch.models.ExactGP):
def __init__(self, train_x, train_y, likelihood):
super().__init__(train_x, train_y, likelihood)
self.mean_module = gpytorch.means.ZeroMean()
self.covar_module =
gpytorch.kernels.ScaleKernel(
gpytorch.kernels.RBFKernel()) #1
def forward(self, x):
... #2
A ScaleKernel wrapped around an RBFKernel separates output scaling from the RBF length scale, making both covariance controls explicit. section 3.4.1
Use length scales to match variation across inputs:
- Use one length scale for a one-dimensional objective; optimize it as a covariance hyperparameter. section 3.4.1
- Use automatic relevance determination (ARD) when dimensions vary at different rates; assign and optimize one length scale per dimension. section 3.4.3 section 3.6
- Interpret a learned smaller value as faster variation along that dimension and a larger value as smoother behavior. section 3.4.3
Toy objectives
A two-dimensional Ackley objective has more valleys along the y-axis than the x-axis, so it requires different length scales. An SVM accuracy surface changes faster with γ than with penalty parameter c; an ARD GP should therefore learn significantly different values. section 3.4.3 section 3.5
