A hyperparameter controls a model’s behavior or structure rather than representing an observed system input. In Gaussian process regression, sigma is a hyperparameter controlling surrogate smoothness: larger values produce smoother surfaces. section 6.2.1
Choosing sigma in Gaussian process regression
A kernel weights measurements by distance after scaling that distance with sigma. With parameters 0.5 and 0.0, measurements 1.52 and 1.21, and sigma 0.25, the weighted model estimates 1.365 at 0.25 and 1.448 at 0.4. Leave-one-out cross-validation can select sigma by maximizing prediction accuracy for each held-out measurement. section 6.2.1
Listing 6.3 GPR with a weighted average
class GPR2:
def __init__(self, parameters, measurements, sigma):
self.x = parameters
self.y = np.array(measurements)
self.sigma = sigma
self.mean_y = self.y.mean()
self.y -= self.mean_y #1
def kernel(self, x1, x2): #2
distance_squared = ((x1-x2)**2).sum()
return np.exp( -distance_squared/(2*self.sigma**2) )
def estimate(self, query_parameter):
weights = [
self.kernel(x, query_parameter)
for x in self.x
] #3
weights = np.array(weights)
weights = weights / weights.sum() #4
return self.mean_y + weights @ self.y #5
Inspect the compact Python implementation that stores sigma, computes the kernel, normalizes weights, and estimates from weighted residuals. section 6.2.1
