Amortized inference

As explained in Machine Learning Algorithms in Depth

Amortized variational inference learns a shared inference function that predicts variational posterior parameters from observations instead of optimizing separate variational parameters for each observation.

Amortized variational inference replaces separate approximate-posterior parameters for each observation with a shared parameterized function. For an observation x and latent variable z, a neural network can map x to the mean and variance of an approximate posterior for z. Training adjusts the network parameters rather than a separate parameter set for every x. section 11.2

The shared mapping changes how inference is carried out:

• It lets observations use a learned inference function instead of requiring separate parameter optimization. section 11.2
• Its generalization gap depends on the capacity of the chosen neural network. section 11.2

Neural-network parameterization

A variational autoencoder uses an encoder to map an observation to latent-distribution parameters. A related application uses a multilayer perceptron in a mixture density network to parameterize a Gaussian mixture model. The mixture model illustrates neural parameterization; it should not be confused with the approximate posterior in every amortized-inference setup. section 11.2 section 11.7

Figure 11.6 A simple Bayesian network. Query 1 is a subquery of query 2.

The Bayesian-network figure shows one query nested within another, illustrating the motivation to reuse earlier inference when answering a related query. section 11.2

How Machine Learning Algorithms in Depth covers it

How deep it goes

Defines the parameter mapping, explains reuse and a capacity limitation, and introduces a neural-network application without deriving an inference objective.

Examples

Uses nested Bayesian-network queries and a multilayer-perceptron mixture density network as illustrations.

What it assumes

Assumes familiarity with observations, latent variables, approximate posterior distributions, and neural networks.

What you can do after

Evaluate/understand it as shared prediction of approximate-posterior parameters and recognize its neural-network application.

How it connects

Connects variational inference to inference networks, variational autoencoders, mixture density networks, and Gaussian mixture models.

liveBookmanning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Related concepts

• language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• mixture density networkused by · A mixture density network is a neural network that predicts the parameters of a conditional mixture probability distribution, optionally at each sequence step.
• stochastic gradient descentuses · Stochastic gradient descent is an iterative optimization method that updates parameters using gradients estimated from individual examples or randomly selected small batches.
• approximate distributionuses · An approximate posterior distribution is a tractable probability distribution selected or optimized to represent an intractable posterior distribution over unknown quantities.
• neural networkuses · A neural network is a computational model of interconnected, parameterized units that learns transformations from inputs to outputs.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.