Kullback–Leibler divergence
Kullback–Leibler divergence is a nonnegative, directed measure of the information lost when one probability distribution approximates another, computed as an expected log ratio under the first distribution.
Reinforcement Learning from Human Feedback
Forward KL divergence is KL divergence from a target distribution to an approximating distribution, with expectation taken under the target distribution.
- How it is used
- Regularizes RLHF policy updates against a reference policy and characterizes SFT and distillation objectives by sampling direction.
- How deep it goes
- Defines both KL directions, derives forward-KL links to SFT, and gives a sampled language-model implementation while deferring mathematical definitions to an appendix.
- Examples
- Uses teacher–student cross-entropy, generated language-model sequences, reference-policy comparisons, and train-versus-test reward-model behavior.
- What you can do after
- Apply it via language-model log-probabilities to estimate policy-to-reference KL and tune its regularization strength.
Read how Reinforcement Learning from Human Feedback explains Kullback–Leibler divergence →
Graph Neural Networks in Action
A measure of the difference between two probability distributions, quantifying the information cost of approximating one with another.
- How it is used
- Regularizes learned latent distributions in graph generative models and neural relational inference alongside reconstruction losses.
- How deep it goes
- Explains distribution overlap and reconstruction trade-offs, then shows PyG and explicit loss implementations without a rigorous derivation.
- Examples
- Uses a fair-coin comparison, Amazon Products edge prediction, and QED-guided molecular graph generation.
- What you can do after
- Apply it via PyTorch Geometric’s VGAE loss or an explicit molecular-graph loss, then validate reconstruction and downstream results.
Read how Graph Neural Networks in Action explains Kullback–Leibler divergence →
Math and Architectures of Deep Learning
A loss term based on Kullback-Leibler divergence that penalizes differences between probability distributions, such as an encoder's latent distribution and a chosen prior.
- How it is used
- Regularizes VAE latent distributions toward a chosen prior alongside reconstruction loss.
- How deep it goes
- Defines core properties, derives Gaussian cases, and connects KL minimization to ELBO and VAE losses.
- Examples
- Compares Gaussian means and spreads, then trains a VAE and autoencoder on MNIST images.
- What you can do after
- Apply it via PyTorch to compare distributions and construct a VAE regularization loss.
Read how Math and Architectures of Deep Learning explains Kullback–Leibler divergence →
Learn Generative AI with PyTorch
A divergence measure used as a regularization term to encourage a latent-variable distribution to resemble a normal distribution.
- How it is used
- Regularizes a VAE encoder against a standard normal prior while the model learns to reconstruct face images.
- How deep it goes
- Defines distribution divergence, explains its zero point for normal latents, and places it in the training loop without deriving the formula.
- Examples
- Uses a 100-dimensional face-image VAE and grouped face encodings that are averaged for encoding arithmetic.
- What you can do after
- Build it from scratch in a VAE by calculating KL loss alongside reconstruction loss during training.
Read how Learn Generative AI with PyTorch explains Kullback–Leibler divergence →
Build a Reasoning Model (From Scratch)
A measure of how different two probability distributions are, used here for soft distillation.
- How it is used
- Uses distribution matching as the training objective for soft distillation.
- How deep it goes
- Defines the metric and contrasts its required probability information with hard distillation, without deriving its formula.
- Examples
- Uses teacher-generated tokens, full vocabulary distributions, and alternative token probabilities as contrasting supervision examples.
- What you can do after
- Apply it via soft distillation by training a student to match teacher token distributions when logits or log probabilities are available.
Read how Build a Reasoning Model (From Scratch) explains Kullback–Leibler divergence →
Machine Learning Algorithms in Depth
Kullback-Leibler divergence is a nonnegative measure of the discrepancy between two probability distributions.
- How it is used
- Measures how closely a tractable distribution approximates a target posterior, making variational inference an optimization problem.
- How deep it goes
- Defines directional KL, works through Gaussian parameters, and relates its penalty to variational free energy and optimization.
- Examples
- Uses Gaussian approximations, a two-mode target, a wireless channel, and an LSTM-VAE anomaly detector.
- What you can do after
- Evaluate/understand directional approximation behavior and compute KL divergence for two univariate Gaussians as an exercise.
Read how Machine Learning Algorithms in Depth explains Kullback–Leibler divergence →
Evolutionary Deep Learning: Genetic algorithms and neural networks
A divergence-based measure comparing probability distributions through differences in their information content.
- How it is used
- Measures distributional loss alongside reconstruction loss when training a variational autoencoder.
- How deep it goes
- Defines divergence as statistical distance and shows its implementation formula, while deferring deeper treatment of statistical distance and loss calculations.
- Examples
- Uses a Keras VAE implementation and compares normalized training losses produced by MSE and binary cross entropy.
- What you can do after
- Apply it via Keras by adding a KL term to a VAE’s reconstruction loss.
Causal AI
Kullback–Leibler divergence quantifies the difference between two probability distributions.
- How it is used
- Uses distributional difference to train a variational autoencoder when the target posterior is inaccessible.
- How deep it goes
- Defines the measure operationally and connects it to negative ELBO, but does not derive its mathematical formula.
- Examples
- Uses Q(Z|X, is-handwritten, digit) and P(Z|image, is-handwritten, digit) as approximating and target distributions.
- What you can do after
- Apply it via Pyro's Trace_ELBO within stochastic variational inference training.
Data Without Labels
Kullback-Leibler divergence is a directed measure of the difference between two probability distributions based on their relative information content.
- How it is used
- Uses distribution differences to guide t-SNE’s preservation of neighborhood relationships during dimensionality reduction.
- How deep it goes
- Defines zero divergence and applies the measure as a gradient-descent cost, without deriving its mathematical formula.
- Examples
- Uses conditional similarities for nearby and distant points, plus Gaussian-versus-Cauchy distribution comparisons.
- What you can do after
- Evaluate how closely low-dimensional similarity probabilities match high-dimensional probabilities in t-SNE.
Read how Data Without Labels explains Kullback–Leibler divergence →
Bayesian Optimization in Action
Kullback–Leibler divergence is a nonnegative statistical divergence that quantifies how one probability distribution differs from another.
- How it is used
- Measures posterior-distribution mismatch when selecting inducing points for scalable variational Gaussian processes.
- How deep it goes
- Defines nonnegativity and matching behavior, then applies the measure to GP posteriors while deferring detailed mathematical computation.
- Examples
- Uses Euclidean distance between points and posterior Gaussian processes trained on inducing points or complete datasets.
- What you can do after
- Evaluate/understand how distributional similarity guides approximate Gaussian-process training through ELBO optimization.
Read how Bayesian Optimization in Action explains Kullback–Leibler divergence →
Optimization Algorithms: AI techniques for design, planning, and control problems
Kullback–Leibler divergence is a metric quantifying the dissimilarity between two probability distributions.
- How it is used
- Constrains PPO policy updates by penalizing excessive divergence from the old policy.
- How deep it goes
- Defines the integral and zero condition, but stops short of deriving the measure or detailing numerical computation.
- Examples
- Uses Gaussian distributions with different means and variances as a visual comparison.
- What you can do after
- Evaluate how far an updated policy departs from an earlier policy within PPO-style optimization.
Deep Learning with R, Third Edition
Kullback–Leibler divergence is a measure of how one probability distribution differs from a reference probability distribution.
- How it is used
- Regularizes variational autoencoders by nudging encoder distributions toward a normal latent distribution.
- How deep it goes
- Defines the training role and intended distributional effect, but does not derive the divergence mathematically.
- Examples
- Uses z_mean, z_log_var, latent sampling, decoding, reconstruction loss, and regularization loss as the working example.
- What you can do after
- Understand how distributional regularization shapes a variational autoencoder’s latent space.
Read how Deep Learning with R, Third Edition explains Kullback–Leibler divergence →
Related concepts
- Group Relative Policy Optimizationpart of · Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled outputs or actions.
- t-SNEused by · t-SNE is a nonlinear dimensionality-reduction technique that maps high-dimensional data to a lower-dimensional space while emphasizing local similarities or neighborhood relationships.
- variational inferenceused by · Variational inference is an approximate inference technique that optimizes a tractable family of probability distributions to approximate a target distribution, often a posterior.
- variational Gaussian processused by · A variational Gaussian process is a scalable approximation to a Gaussian process whose posterior is represented through variational inference over a finite set of inducing points.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.