A kernel function K(x, z) compares two inputs. In the dual form of a support vector machine (SVM), data points interact through inner products; replacing those inner products with kernel evaluations lets the classifier use a similarity measure without explicitly constructing feature vectors. Kernels can compare objects such as graphs, strings, or documents. section 5.3
The radial basis function (RBF) kernel used in the SVM implementation is K(x, z) = exp(−γ‖x − z‖²). Its value decreases as squared distance increases; γ controls how strongly distance affects similarity. The implementation computes the squared distance with NumPy, then exponentiates its negative scaled value. section 5.3
A Gaussian process (GP) gives a kernel another job: defining covariance between function values. For inputs x₁, …, xₙ, the covariance matrix has entries Σᵢⱼ = κ(xᵢ, xⱼ), where κ is a positive-definite kernel. For training inputs X and test inputs X*, compute K = κ(X, X), K* = κ(X, X*), and K** = κ(X*, X*); conditioning their joint Gaussian distribution gives the predictive posterior. section 6.5
One similarity measure, two uses
An RBF kernel supplies pairwise similarities to an SVM optimized with CVXOPT. In GP regression, an RBF kernel supplies training and test covariance matrices for posterior prediction. The GP implementation computes pairwise squared distances with SciPy and uses a length scale L in exp(−‖x/L − z/L‖²/2). section 5.3 section 6.5
Listing 6.4 Gaussian process regression
import numpy as np
import matplotlib.pyplot as plt
from scipy.spatial.distance import cdist
np.random.seed(42)
class GPreg:
def __init__(self, X_train, y_train, X_test):
self.L = 1.0
self.keps = 1e-8
self.muFn = self.mean_func(X_test)
self.Kfn = self.kernel_func(X_test, X_test) + 1e-15*np.eye(
np.size(X_test))
self.X_train = X_train
self.y_train = y_train
self.X_test = X_test
def mean_func(self, x):
muFn = np.zeros(len(x)).reshape(-1,1)
return muFn
def kernel_func(self, x, z):
sq_dist = cdist(x/self.L, z/self.L, 'euclidean')**2
Kfn = 1.0 * np.exp(-sq_dist/2)
return Kfn
def compute_posterior(self):
K = self.kernel_func(self.X_train, self.X_train)
Ks = self.kernel_func(self.X_train, self.X_test)
Kss = self.kernel_func(self.X_test, self.X_test) + self
.keps*np.eye(np.size(self.X_test))
Ki = np.linalg.inv(K) #1
postMu = self.mean_func(self.X_test) + np.dot(np.transpose(Ks),
np.dot(Ki, (self.y_train - self.mean_func(self.X_train))))
postCov = Kss - np.dot(np.transpose(Ks), np.dot(Ki, Ks))
self.muFn = postMu
self.Kfn = postCov
return None
def generate_plots(self, X, num_samples=3):
plt.figure()
for i in range(num_samples):
fs = self.gauss_sample(1)
plt.plot(X, fs, '-k')
#plt.plot(self.X_train, self.y_train, 'xk')
mu = self.muFn.ravel()
S2 = np.diag(self.Kfn)
plt.fill(np.concatenate([X, X[::-1]]), np.concatenate([mu –
2*np.sqrt(S2), (mu + 2*np.sqrt(S2))[::-1]]), alpha=0.2, fc='b')
plt.show()
def gauss_sample(self, n): #2
A = np.linalg.cholesky(self.Kfn)
Z = np.random.normal(loc=0, scale=1, size=(len(self.muFn),n))
S = np.dot(A,Z) + self.muFn #3
return S
def main():
# generate noise-less training data
X_train = np.array([-4, -3, -2, -1, 1])
X_train = X_train.reshape(-1,1)
y_train = np.sin(X_train)
# generate test data
X_test = np.linspace(-5, 5, 50)
X_test = X_test.reshape(-1,1)
gp = GPreg(X_train, y_train, X_test)
gp.generate_plots(X_test,3) #4
gp.compute_posterior()
gp.generate_plots(X_test,3) #5
if __name__ == "__main__":
main()
The GP regression listing shows how kernel_func constructs covariance matrices for training and test inputs. section 6.5
