Approximate distribution

As explained in Machine Learning Algorithms in Depth

An approximate posterior distribution is a tractable distribution used to approximate the conditional distribution of latent variables given observed data.

An approximate distribution q(x) stands in for a target posterior p(x) when the target is difficult to compute directly. Variational inference chooses q from a tractable family with trainable parameters, then adjusts those parameters to make q close to p. Unlike a sample-based approximation, the result is itself a distribution that can be evaluated or optimized. section 3.0

One objective is reverse KL divergence, KL(q‖p) = E_q[log(q(x)/p(x))]. Minimizing it turns posterior approximation into an optimization problem. KL is nonnegative and reaches zero when q equals p. The choice of direction matters: reverse KL tends to concentrate on one mode of the target, while forward KL tends to cover more of its support. section 3.1

A normal approximation

Let the target be p(x) = N(0, 1) and the approximation be q(x) = N(μ, 4). Holding q’s variance fixed and varying μ from −4 to 4 changes KL(q‖p); it is smallest at μ = 0, where the two means agree. Matching means does not make the distributions identical, because their variances still differ. section 3.1

Figure 3.2 Forward KL (left) q(x) overestimates the support, while reverse KL (right) q(x) locks onto a mode.

Compare how the direction of KL divergence changes the approximation: forward KL spreads across support, whereas reverse KL locks onto a mode. section 3.1

The tractable family determines what q can represent. A mean-field approximation makes q fully factorized, speeding computation but restricting dependence between variables. Its parameters can be learned by maximizing an evidence lower bound (ELBO). In amortized variational inference, a parameterized function instead maps observations to approximate-posterior parameters. Approximate distributions also appear when deriving a lower bound for mutual-information maximization. section 3.0 section 3.6 section 11.7 section 3.4

How Machine Learning Algorithms in Depth covers it

How deep it goes

Defines the approximation and develops KL-based optimization, with mean-field factorization and an ELBO objective.

Examples

Compares normal distributions while varying the approximate mean and contrasts forward-KL support coverage with reverse-KL mode selection.

What it assumes

Assumes familiarity with probability distributions and posterior inference, while introducing the variational approximation and its KL objective.

What you can do after

Evaluate/understand how a tractable posterior approximation is selected and optimized, including the effect of KL direction.

How it connects

Connects variational inference to KL divergence, mean-field approximation, the evidence lower bound, and amortized variational inference.

liveBookmanning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Causal AIA variational distribution is a parameterized probability distribution chosen to approximate a target distribution in variational inference.
• 

  liveBookmanning.com
  Bayesian Optimization in ActionA variational distribution is an optimized probability distribution that approximates an intractable posterior distribution in variational inference.

Compare how 3 books explain approximate distribution →

Related concepts

• variational inferenceused by · Variational inference is an approximate inference technique that optimizes a tractable family of probability distributions to approximate a target distribution, often a posterior.
• amortized inferenceused by · Amortized inference is a technique that learns a reusable mapping from observations to approximate posterior distributions or their parameters.
• latent Dirichlet allocationused by · Latent Dirichlet allocation is a generative probabilistic topic model representing documents as mixtures of latent topics and topics as distributions over words.
• posterior probabilitycontrasts with · A posterior probability or distribution expresses the probability assigned to an event, hypothesis, or unknown quantity conditional on observed evidence and prior information.
• variational Gaussian processhas part · A variational Gaussian process is a scalable approximation to a Gaussian process whose posterior is represented through variational inference over a finite set of inducing points.
• guide functionproduced by · A guide function is a Pyro stochastic function that defines the variational approximating process used for inference.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.