How deeply does Data Without Labels cover contrastive divergence?

Explains stochastic gradient estimation through alternating Gibbs updates, without detailing equations or implementation code.

Contrastive divergence

As explained in Data Without Labels

Contrastive divergence is a stochastic approximation technique that estimates the gradient of a generative model's log-likelihood using short Gibbs-sampling chains.

Contrastive divergence is a stochastic approximation technique for estimating a generative model’s log-likelihood gradient. It makes Boltzmann learning more practical by replacing its computationally expensive basic procedure with a faster approximation. section 8.8.2 section 8.9.1

A contrastive-divergence update follows this pattern:

Start with a sample from the visible layer. section 8.9.1
Use Gibbs sampling to update the hidden layer and then the visible layer iteratively. section 8.9.1
Use the resulting stochastic estimate to adjust weights toward modeling the input-data distribution. section 8.9.1

Layer-wise RBM pretraining

During DBN pretraining, train each RBM independently with contrastive divergence. Earlier layers capture low-level features, while later layers learn increasingly abstract representations without labeled data. section 8.9.1

How Data Without Labels covers it

liveBook manning.com

Read Data Without Labels on liveBook Get the book at manning.com

Related concepts

gibbs samplinguses · Gibbs sampling is a Markov chain Monte Carlo technique that repeatedly samples each variable from its conditional distribution given the others.
restricted Boltzmann machineused by · A restricted Boltzmann machine is a generative probabilistic neural-network model whose visible and hidden units are connected across layers but not within a layer.
Boltzmann learning ruleused by · The Boltzmann learning rule is a learning procedure that updates connection weights to reduce the energy of a stochastic neural network while matching data and model distributions.

This explainer was generated by Manning's knowledge graph from the text of Data Without Labels. The section links point to the parts of the book it draws on.