liveBook · Manning

Kernel function

As explained in Timeless Algorithms, The Foundational Ideas

A function that computes an inner product in a feature space, allowing some learning methods to operate implicitly in that space.

A nonlinear classification problem may become linearly separable when observations are mapped into a richer feature space. But constructing every transformed coordinate can be impractical. A kernel function calculates the inner product in that space directly from the original observations: K(x, z) = ϕ(x) · ϕ(z), where ϕ is the feature transformation. This substitution is the kernel trick. section 8.6 section 8.6.1

In a support vector machine (SVM), the decision score can be written as f(x) = ∑ᵢ∈SV αᵢyᵢK(xᵢ, x) + b. Here xᵢ are support vectors, yᵢ their labels, αᵢ their fitted coefficients, and b the bias. The kernel supplies the transformed dot products; it does not replace the other components of the decision function. A hyperplane linear in the implicit feature space can correspond to a nonlinear boundary in the original space. section 8.6.1 section 8.6

The same dot product without transformed coordinates

Let ϕ(x) = (x, x²), so K(x, z) = xz + (xz)². For temperature readings 3 and 6, the transformed observations are (3, 9) and (6, 36), whose dot product is 18 + 324 = 342. The kernel gives K(3, 6) = 18 + 18² = 342 directly from the original readings. Both routes give the same value. section 8.6.1

Common kernels offer different ways to compare observations:

  • Linear: K(x, z) = x · z. It uses the supplied feature space and produces a flat boundary in that space. section 8.6.2
  • Polynomial: K(x, z) = (x · z + c)ᵈ. The degree d controls the powers and interactions represented, while c affects the influence of lower-degree terms. section 8.6.2
  • Radial basis function (RBF): K(x, z) = exp(−γ||x − z||²). Gamma controls how locally the influence of a support vector is concentrated. section 8.6.2 section 8.6.3
Figure 8.7 The same synthetic two-class data fitted with linear, third-degree polynomial, and RBF kernels. The linear kernel produces a straight boundary, the polynomial kernel captures higher-order interactions through a curved boundary, and the RBF kernel produces a localized nonlinear boundary. The SVM framework remains unchanged; only the kernel used to compare observations differs.

The same synthetic data produce a straight, curved, or localized boundary under linear, polynomial, and RBF kernels, respectively. section 8.6.2

A kernel defines a family of possible boundaries, not the fitted boundary by itself. Training data, support vectors, the soft-margin penalty C, and kernel parameters also matter. Higher polynomial degree or larger γ can capture more complex patterns but may fit training irregularities too closely. Evaluate candidate kernels and parameter combinations with cross-validation rather than training accuracy alone. Kernel SVMs can also become computationally expensive as the training set grows. section 8.6.2 section 8.6.3 section 8.8.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Derives the kernel substitution in the support-vector decision function and compares kernel families, parameters, and model-selection tradeoffs.
Examples
Calculates a transformed temperature dot product and compares linear, polynomial, and RBF boundaries on synthetic two-class data.
What it assumes
Builds on dot products and support-vector decision functions, then derives kernel comparisons from explicit feature transformations.
What you can do after
Evaluate kernel choices and parameter settings using their boundary behavior and cross-validation rather than training accuracy alone.
How it connects
Connects feature transformations, inner products, support vector machines, maximum-margin separation, and cross-validation.

Also explained in

Compare how 6 books explain kernel function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.