Kernel function

As explained in Grokking Machine Learning

A mathematical function that represents an inner product in a transformed feature space, allowing a model to learn nonlinear decision boundaries.

A kernel function measures similarity between inputs and represents their inner product in a transformed feature space. This lets a model find nonlinear decision boundaries by separating points in a richer space. section 11.3

For an SVM, the kernel method adds features so data that cannot be separated by a line may become separable by a plane. Projecting that boundary back to the original space can produce a curve. A radial basis function (RBF) kernel gives higher similarity to nearby points and lower similarity to distant ones; polynomial kernels offer another option. section 11.3

Similarity with an RBF kernel

For two points at distance √2, the example similarity is e⁻², or about 0.135. Adding similarity features for the data points can make a one-dimensional dataset with a middle point of one class and two outer points of another linearly separable. section 11.3

Figure 11.17 The kernel method. Step 1: We start with a dataset that is not linearly separable. Step 2: Then we raise each point by a distance that is the square of its distance to the origin. This creates a paraboloid. Step 3: Now the triangles are high, whereas the squares are low. We proceed to separate them with a plane at height 1. Step 4. We project everything down. The intersection between the paraboloid and the plane creates a circle. The projection of this circle gives us the circular boundary of our classifier. (Source: Image created with the assistance of Grapher™ from Golden Software, LLC; https://www.goldensoftware.com/products/grapher).

See how raising points by their squared distance makes a dataset separable by a plane, whose projection becomes a circular boundary. section 11.3

Try the XOR dataset: choose a kernel that makes it linearly separable, then sketch the resulting SVM boundary. section 11.5

How Grokking Machine Learning covers it

How deep it goes

Explains similarity, feature mapping, and RBF and polynomial examples, while deferring the full theory of kernels and feature maps.

Examples

Uses XOR, circular and one-dimensional datasets to illustrate polynomial and RBF kernel classification.

What it assumes

Builds on linear separability and support vector machines, and explains kernel geometry through added features and similarity.

What you can do after

Apply via SVC by selecting a polynomial or RBF kernel and setting its degree or gamma.

How it connects

Connects kernels to support vector machines, feature maps, linear classifiers, and radial basis and polynomial functions.

liveBookmanning.com

Read Grokking Machine Learning on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Causal Inference for Data ScienceA kernel is a weighting function that assigns observation weights according to distance from a target point.
• 

  liveBookmanning.com
  Experimentation for EngineersA kernel function assigns a similarity or weight to an observation based on its distance from another input.
• 

  liveBookmanning.com
  Machine Learning Algorithms in DepthA positive-definite function that measures similarity between pairs of inputs and defines an associated feature-space inner product.

Compare how 4 books explain kernel function →

Related concepts

• feature mapproduces · A feature map is a spatially organized array of neural-network activations representing the presence or strength of learned features at different locations.
• covariance functionequivalent to · A covariance function, also called a kernel, specifies the covariance between function values at pairs of inputs in a Gaussian process.
• kernel density estimateused by · A kernel density estimate is a nonparametric probability-density estimate formed by combining localized kernel functions centered on observed data points.
• similarity metrickind of · A similarity measure is a rule that quantifies or characterizes how alike two objects or representations are according to selected properties.
• radial basis function kernelincludes · A radial basis function kernel is a similarity function whose value depends on the distance between inputs and typically decreases as that distance increases.
• support vector machineused by · A support vector machine is a supervised-learning model that uses support vectors to determine a maximum-margin decision boundary or regression function, optionally using kernels to model nonlinear relationships.

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning. The section links point to the parts of the book it draws on.