liveBook · Manning

Kullback–Leibler divergence

Also called KL divergence, forward KL

Kullback–Leibler divergence is a nonnegative, generally asymmetric measure of the difference between probability distributions, defined as the expected log probability ratio under one distribution.

How 19 Manning books explain it

Reinforcement Learning from Human Feedback

Forward KL divergence is KL divergence from a target distribution to an approximating distribution, with expectation taken under the target distribution.

How it is used
Uses KL divergence to constrain policy drift and compare post-training objectives.
How deep it goes
Derives KL-based objectives and explains a sample-based sequence implementation.
Examples
Uses a policy-reference training penalty and teacher–student distillation to illustrate KL directions.
What you can do after
Evaluate how KL direction, penalty strength, and sampling affect post-training optimization.

Read how Reinforcement Learning from Human Feedback explains Kullback–Leibler divergence →

Graph Neural Networks in Action

A measure of the difference between two probability distributions, quantifying the information cost of approximating one with another.

How it is used
Regularizes learned latent distributions in graph generative models and neural relational inference alongside reconstruction losses.
How deep it goes
Explains distribution overlap and reconstruction trade-offs, then shows PyG and explicit loss implementations without a rigorous derivation.
Examples
Uses a fair-coin comparison, Amazon Products edge prediction, and QED-guided molecular graph generation.
What you can do after
Apply it via PyTorch Geometric’s VGAE loss or an explicit molecular-graph loss, then validate reconstruction and downstream results.

Read how Graph Neural Networks in Action explains Kullback–Leibler divergence →

Math and Architectures of Deep Learning

A loss term based on Kullback-Leibler divergence that penalizes differences between probability distributions, such as an encoder's latent distribution and a chosen prior.

How it is used
Regularizes VAE latent distributions toward a chosen prior alongside reconstruction loss.
How deep it goes
Defines core properties, derives Gaussian cases, and connects KL minimization to ELBO and VAE losses.
Examples
Compares Gaussian means and spreads, then trains a VAE and autoencoder on MNIST images.
What you can do after
Apply it via PyTorch to compare distributions and construct a VAE regularization loss.

Read how Math and Architectures of Deep Learning explains Kullback–Leibler divergence →

Learn Generative AI with PyTorch

A divergence measure used as a regularization term to encourage a latent-variable distribution to resemble a normal distribution.

How it is used
Regularizes a variational autoencoder’s latent distributions toward a standard normal prior alongside reconstruction loss.
How deep it goes
Defines divergence and explains its normal-prior minimum across 100 latent dimensions, without deriving the formula.
Examples
Uses face-image reconstruction, sampled face generation, and glasses-related encoding arithmetic and interpolation.
What you can do after
Evaluate and understand how KL regularization contributes to VAE training and latent-space manipulation.

Read how Learn Generative AI with PyTorch explains Kullback–Leibler divergence →

Build a Reasoning Model (From Scratch)

A measure of how different two probability distributions are, used here for soft distillation.

How it is used
Uses distribution matching as the training objective for soft distillation.
How deep it goes
Defines the metric and contrasts its required probability information with hard distillation, without deriving its formula.
Examples
Uses teacher-generated tokens, full vocabulary distributions, and alternative token probabilities as contrasting supervision examples.
What you can do after
Apply it via soft distillation by training a student to match teacher token distributions when logits or log probabilities are available.

Read how Build a Reasoning Model (From Scratch) explains Kullback–Leibler divergence →

Rearchitecting LLMs

Kullback–Leibler divergence measures how one probability distribution differs from a reference distribution.

How it is used
Uses KL divergence as a student-to-teacher distribution-matching loss for knowledge distillation and knowledge recovery.
How deep it goes
Defines divergence through distribution agreement and implements both direct and temperature-scaled KL loss, without deriving the formula.
Examples
Uses a pruned student matching a teacher’s logits and a compound loss combining task, logit, and hidden-state signals.
What you can do after
Apply it via PyTorch to train a student against teacher probabilities, either directly or as part of a compound loss.

Read how Rearchitecting LLMs explains Kullback–Leibler divergence →

Build a DeepSeek Model (From Scratch)

KL divergence is a measure of the discrepancy between two probability distributions that quantifies information lost when one distribution approximates another.

How it is used
Uses KL to transfer teacher probability distributions and constrain policy drift during reinforcement learning.
How deep it goes
Explains temperature-corrected gradients and a PyTorch loss, while leaving production RL details such as KL scheduling to further work.
Examples
Works through a four-class distillation loss at T=3 and a GRPO objective with a reference-policy penalty.
What you can do after
Apply it via PyTorch to combine teacher-student KL with hard-label cross-entropy.

Read how Build a DeepSeek Model (From Scratch) explains Kullback–Leibler divergence →

Build AI Drug Discovery Pipelines

Kullback-Leibler divergence is a measure of how one probability distribution differs from a reference probability distribution.

How it is used
Regularizes molecular VAE latent distributions so sampling can produce novel molecules from a continuous chemical space.
How deep it goes
Defines the objective and intuition, then stops short of the probabilistic derivation and complete training implementation.
Examples
Uses molecular SMILES, Gaussian latent clouds, a standard-normal prior, and the VAE-CYC training pipeline as concrete examples.
What you can do after
Build molecular VAEs that regularize latent distributions and generate molecules by sampling from the prior.

Read how Build AI Drug Discovery Pipelines explains Kullback–Leibler divergence →

Machine Learning Algorithms in Depth

Kullback-Leibler divergence is a nonnegative measure of the discrepancy between two probability distributions.

How it is used
Measures how closely a tractable distribution approximates a target posterior, making variational inference an optimization problem.
How deep it goes
Defines directional KL, works through Gaussian parameters, and relates its penalty to variational free energy and optimization.
Examples
Uses Gaussian approximations, a two-mode target, a wireless channel, and an LSTM-VAE anomaly detector.
What you can do after
Evaluate/understand directional approximation behavior and compute KL divergence for two univariate Gaussians as an exercise.

Read how Machine Learning Algorithms in Depth explains Kullback–Leibler divergence →

Grokking Bayes

KL divergence is a nonnegative measure of the directed difference between two probability distributions, equal to zero when they are identical.

How it is used
Measures the gap between an approximating distribution and a Bayesian posterior during variational inference.
How deep it goes
Defines nonnegativity and mismatch behavior, then connects minimization to ELBO optimization and variance underestimation.
Examples
Uses a balloon-and-jelly analogy, a target Beta distribution, and visual demonstrations of mode-seeking and variance underestimation.
What you can do after
Evaluate variational approximations by relating ELBO improvement to decreasing KL divergence and recognizing mode-seeking failure.

Read how Grokking Bayes explains Kullback–Leibler divergence →

Retrieval Augmented Generation, The Foundational Ideas

Kullback-Leibler divergence is a measure of how one probability distribution differs from a reference distribution.

How it is used
Aligns retriever scores with the language model’s estimate of which retrieved documents help answer a query.
How deep it goes
Explains distribution construction, PyTorch implementation, optimization, and loss combination, but does not derive KL mathematically.
Examples
Demonstrates AWS operating-margin passages, reader log-probabilities, retriever distributions, and falling KL during optimization.
What you can do after
Apply it via PyTorch to train a retriever against reader-derived probability targets.

Read how Retrieval Augmented Generation, The Foundational Ideas explains Kullback–Leibler divergence →

Evolutionary Deep Learning: Genetic algorithms and neural networks

A divergence-based measure comparing probability distributions through differences in their information content.

How it is used
Measures distributional loss alongside reconstruction loss when training a variational autoencoder.
How deep it goes
Defines divergence as statistical distance and shows its implementation formula, while deferring deeper treatment of statistical distance and loss calculations.
Examples
Uses a Keras VAE implementation and compares normalized training losses produced by MSE and binary cross entropy.
What you can do after
Apply it via Keras by adding a KL term to a VAE’s reconstruction loss.

Read how Evolutionary Deep Learning: Genetic algorithms and neural networks explains Kullback–Leibler divergence →

Causal AI

Kullback–Leibler divergence quantifies the difference between two probability distributions.

How it is used
Uses distributional difference to train a variational autoencoder when the target posterior is inaccessible.
How deep it goes
Defines the measure operationally and connects it to negative ELBO, but does not derive its mathematical formula.
Examples
Uses Q(Z|X, is-handwritten, digit) and P(Z|image, is-handwritten, digit) as approximating and target distributions.
What you can do after
Apply it via Pyro's Trace_ELBO within stochastic variational inference training.

Read how Causal AI explains Kullback–Leibler divergence →

Data Without Labels

Kullback-Leibler divergence is a directed measure of the difference between two probability distributions based on their relative information content.

How it is used
Uses distribution differences to guide t-SNE’s preservation of neighborhood relationships during dimensionality reduction.
How deep it goes
Defines zero divergence and applies the measure as a gradient-descent cost, without deriving its mathematical formula.
Examples
Uses conditional similarities for nearby and distant points, plus Gaussian-versus-Cauchy distribution comparisons.
What you can do after
Evaluate how closely low-dimensional similarity probabilities match high-dimensional probabilities in t-SNE.

Read how Data Without Labels explains Kullback–Leibler divergence →

LLM Evaluation and Alignment, The Foundational Ideas

Kullback-Leibler divergence is a measure of how one probability distribution differs from a reference probability distribution.

How it is used
Controls policy drift during PPO-based reinforcement learning from human feedback.
How deep it goes
Defines distributional difference and explains its reward penalty, but does not derive the divergence mathematically.
Examples
Uses repetitive token sequences, gibberish, and teacher–student token-probability matching as concrete contexts.
What you can do after
Apply it via PPO to balance reward-model gains against changes from a reference policy.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains Kullback–Leibler divergence →

Bayesian Optimization in Action

Kullback–Leibler divergence is a nonnegative statistical divergence that quantifies how one probability distribution differs from another.

How it is used
Measures posterior-distribution mismatch when selecting inducing points for scalable variational Gaussian processes.
How deep it goes
Defines nonnegativity and matching behavior, then applies the measure to GP posteriors while deferring detailed mathematical computation.
Examples
Uses Euclidean distance between points and posterior Gaussian processes trained on inducing points or complete datasets.
What you can do after
Evaluate/understand how distributional similarity guides approximate Gaussian-process training through ELBO optimization.

Read how Bayesian Optimization in Action explains Kullback–Leibler divergence →

Optimization Algorithms: AI techniques for design, planning, and control problems

Kullback–Leibler divergence is a metric quantifying the dissimilarity between two probability distributions.

How it is used
Constrains PPO policy updates by penalizing excessive divergence from the old policy.
How deep it goes
Defines the integral and zero condition, but stops short of deriving the measure or detailing numerical computation.
Examples
Uses Gaussian distributions with different means and variances as a visual comparison.
What you can do after
Evaluate how far an updated policy departs from an earlier policy within PPO-style optimization.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains Kullback–Leibler divergence →

LLM Customization and Fine-Tuning

A measure of how one probability distribution differs from a reference probability distribution.

How it is used
Frames distribution mismatch as a distillation objective for aligning student outputs with a teacher.
How deep it goes
Defines nonnegative, asymmetric divergence and explains its training effects without deriving the measure.
Examples
Contrasts student behavior on a teacher that produces both JSON and Markdown outputs.
What you can do after
Evaluate forward or reverse KL by balancing output coverage, precision, task requirements, and teacher access.

Read how LLM Customization and Fine-Tuning explains Kullback–Leibler divergence →

Deep Learning with R, Third Edition

Kullback–Leibler divergence is a measure of how one probability distribution differs from a reference probability distribution.

How it is used
Regularizes variational autoencoders by nudging encoder distributions toward a normal latent distribution.
How deep it goes
Defines the training role and intended distributional effect, but does not derive the divergence mathematically.
Examples
Uses z_mean, z_log_var, latent sampling, decoding, reconstruction loss, and regularization loss as the working example.
What you can do after
Understand how distributional regularization shapes a variational autoencoder’s latent space.

Read how Deep Learning with R, Third Edition explains Kullback–Leibler divergence →

Also covered in

What to know first

Related concepts

Questions about Kullback–Leibler divergence

What is Kullback–Leibler divergence?

Kullback–Leibler divergence is a nonnegative, generally asymmetric measure of the difference between probability distributions, defined as the expected log probability ratio under one distribution.

Which Manning books explain Kullback–Leibler divergence?

Reinforcement Learning from Human Feedback, Graph Neural Networks in Action, Math and Architectures of Deep Learning, Learn Generative AI with PyTorch, Build a Reasoning Model (From Scratch), Rearchitecting LLMs, Build a DeepSeek Model (From Scratch), Build AI Drug Discovery Pipelines and 11 more.

What should I know before learning Kullback–Leibler divergence?

Manning books that teach Kullback–Leibler divergence usually introduce probability distribution, objective function, normal distribution and autoencoder first.

What is Kullback–Leibler divergence part of?

Kullback–Leibler divergence is part of Group Relative Policy Optimization. Group Relative Policy Optimization is a reinforcement-learning algorithm that updates a policy using relative rewards or advantages estimated from groups of sampled actions or outputs.

What is Kullback–Leibler divergence also called?

Kullback–Leibler divergence is also called KL divergence or forward KL.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.