Feature extractor model

As explained in Bayesian Optimization in Action

A feature extractor is a model component that transforms structured inputs into informative numerical features for a downstream model.

A feature extractor model transforms structured inputs into informative numerical features for a downstream model. In a neural-network–Gaussian-process combination, the neural network processes x before the GP mean function and kernel. This preserves the GP’s uncertainty-calibrated predictions while allowing flexible learned representations. section 13.3.1

Raw structured inputs, such as image pixels, may provide a poor basis for GP similarity calculations. Multiple nonlinear neural-network layers can instead learn continuous features whose geometry reflects the prediction task. Training learns both the representation and GP parameters by minimizing negative log likelihood with gradient descent. section 13.3.1 section 13.4

A practical feature-extractor model follows this flow:

• Pass each input through sequential linear and nonlinear layers. section 13.3.2
• Produce a low-dimensional representation; the demonstrated network maps 784 input values to two features. section 13.3.2
• Scale extracted values to a bounded range before applying the GP mean and covariance modules. section 13.3.2
• Optimize feature-extractor weights alongside GP and likelihood parameters. section 13.3.1

MNIST features

A four-layer PyTorch extractor maps flattened 28×28 MNIST images through widths 1000, 500, and 50 to two output features. After training with a GP, examples sharing a digit label cluster together and form a smooth label gradient. Such smooth representations are easier for the GP to learn than randomly swapped labels. section 13.3.2 section 13.3.2

Figure 13.10 Combining a neural network with a GP. The neural network processes the structured data input x first and then feeds the output to the mean function and the kernel of the GP.

Trace the data path from structured input through the neural network into the GP mean and kernel. section 13.3.1

How Bayesian Optimization in Action covers it

How deep it goes

Defines the transformation, architecture, scaling, forward pass, and joint training, but defers broader implementation details.

Examples

Uses a four-layer MNIST extractor, two-dimensional feature plots, smooth-versus-rough label representations, and a test mean-absolute-error result.

What it assumes

Assumes familiarity with neural-network layers and Gaussian-process components while introducing the feature-extraction role from structured data.

What you can do after

Build it from scratch using PyTorch and GPyTorch, then train and inspect a learned representation for GP modeling.

How it connects

Connects neural networks to Gaussian processes, kernels, mean functions, covariance modules, negative log likelihood, and gradient descent.

liveBookmanning.com

Read Bayesian Optimization in Action on liveBook Get the book at manning.com

Related concepts

• neural network-Gaussian process modelused by · A neural network-Gaussian process model combines a neural network transformation of inputs with Gaussian process prediction on the transformed representation.
• Gaussian processused by · A Gaussian process is a probability distribution over functions in which every finite collection of function values has a joint Gaussian distribution.
• neural networkincludes · A neural network is a computational model of interconnected, parameterized units that learns transformations from inputs to outputs.
• structured datauses · Structured data is information organized according to an explicit schema or predictable fields, types, records, or relationships for consistent machine processing.
• feature engineeringimplements · Feature engineering is the creation, transformation, or selection of input features from data to provide useful representations for analysis or modeling.

This explainer was generated by Manning's knowledge graph from the text of Bayesian Optimization in Action. The section links point to the parts of the book it draws on.