liveBook · Manning

Entropy

Entropy is an information-theoretic measure of uncertainty in a probability distribution, defined as the expected value of the negative logarithm of outcome probability.

How 19 Manning books explain it

Grokking Machine Learning

Entropy is an information-theoretic impurity measure that is higher when class probabilities are more diverse and zero for a pure set.

How it is used
Selects classification-tree questions by comparing how mixed their resulting labeled groups are.
How deep it goes
Works through split entropies, size-weighted averages, and information gain, without developing information theory beyond tree splitting.
Examples
Compares age and platform splits for app recommendations and offers a credit-card fraud exercise.
What you can do after
Practice/perform split comparisons using weighted entropy, or apply it via Scikit-Learn’s DecisionTreeClassifier entropy criterion.

Read how Grokking Machine Learning explains entropy →

Bayesian Optimization in Action

Entropy is an information-theoretic measure of the uncertainty or variability of a random variable, defined for a discrete distribution as the negative sum of each event probability times its base-2 logarithm.

How it is used
Uses entropy to choose searches or experiments that most reduce uncertainty about a target quantity.
How deep it goes
Defines Bernoulli entropy, develops expected posterior entropy through search examples, and defers closed-form MES scores to sampling approximations.
Examples
Uses biased coins, a lost remote, a 100-element sorted list, phone-drop testing, and Forrester-function optimization.
What you can do after
Apply it via BoTorch to configure MES policies that approximate expected entropy reduction in Bayesian optimization.

Read how Bayesian Optimization in Action explains entropy →

Timeless Algorithms, The Foundational Ideas

Entropy is a quantitative measure of uncertainty across the possible outcomes of an information source and their probabilities.

How it is used
Uses entropy to quantify uncertainty in communication and to guide learning decisions.
How deep it goes
Defines entropy mathematically and works through bit calculations, compression, and information gain.
Examples
Compares fair and biased coins, a die, and a deck of cards, then calculates a decision-tree split.
What you can do after
Evaluate and understand source uncertainty and compare candidate decision-tree splits using entropy and information gain.

Read how Timeless Algorithms, The Foundational Ideas explains entropy →

Grokking Machine Learning, Second Edition

A measure of uncertainty or impurity in a probability distribution, commonly used to evaluate classification splits.

How it is used
Selects classification-tree questions that make resulting label groups less mixed.
How deep it goes
Defines purity through endpoint values and works through weighted split comparisons, without deriving the entropy formula.
Examples
Uses app recommendations by age or platform and age-threshold questions with weighted entropy values.
What you can do after
Apply it via Scikit-Learn by selecting entropy as a DecisionTreeClassifier criterion and comparing tree splits.

Read how Grokking Machine Learning, Second Edition explains entropy →

Hacking Cryptography

Entropy is a measure of uncertainty in a probability distribution, calculated as the negative sum of each outcome probability multiplied by its logarithmic probability.

How it is used
Quantifies RNG unpredictability for choosing and monitoring cryptographic randomness sources.
How deep it goes
Defines binary-log entropy, calculates biased-coin values, and applies entropy to TRNG health testing and CSPRNG seeding.
Examples
Uses fair and biased coin tosses, including 0.75/0.25 probabilities yielding about 0.81 bits.
What you can do after
Evaluate/understand how probability bias lowers unpredictability and why cryptographic systems require monitored quality entropy.

Read how Hacking Cryptography explains entropy →

Math and Architectures of Deep Learning

A measure of uncertainty or unpredictability in a probability distribution, high when outcomes are nearly equally likely and low when one outcome dominates.

How it is used
Quantifies uncertainty in probability distributions used for Bayesian and unsupervised machine learning.
How deep it goes
Defines discrete entropy with worked image calculations, geometric intuition, and Gaussian derivation, then extends it to conditional entropy.
Examples
Uses rain forecasts, two four-color images, Gaussian distributions, and predicted cat, dog, airplane, and automobile probabilities.
What you can do after
Evaluate distributional uncertainty and compute entropy for discrete distributions or Gaussian distributions with PyTorch.

Read how Math and Architectures of Deep Learning explains entropy →

Grokking Bayes

Entropy is a measure of uncertainty in a random variable, defined as the expected negative logarithm of its probability.

How it is used
Measures uncertainty so candidate observations can be compared by their expected reduction in entropy.
How deep it goes
Defines discrete entropy, explains its behavior, and works through a numerical calculation; notes differential entropy without developing it.
Examples
Uses a uniform prior over 100 integers, a binary-search question, and a Python math.log2 shortcut.
What you can do after
Apply it to compare candidate questions by their expected reduction in uncertainty.

Read how Grokking Bayes explains entropy →

Sutskever's List

Entropy is a thermodynamic quantity related to the number of microscopic configurations compatible with a macroscopic state and to the dispersal of energy and matter.

How it is used
Frames entropy as the monotonic thermodynamic backdrop for explaining why complexity can rise and fall in a closed system.
How deep it goes
Defines entropy through disorder, energy and matter dispersal, and compatible microscopic configurations; illustrates its increase but derives no thermodynamic formulas.
Examples
Uses cream mixing in coffee, contrasting layered beginnings, structured swirls, and a uniform endpoint.
What you can do after
Understand why rising entropy need not mean rising visible or algorithmic complexity.

Read how Sutskever's List explains entropy →

Real-World Cryptography

Entropy is a measure of the uncertainty or unpredictability contained in a random source or secret.

How it is used
Guides the secure generation of cryptographic secrets and random values.
How deep it goes
Defines high and low entropy, then follows randomness through OS seeding, APIs, and early-boot failure modes.
Examples
Uses /dev/urandom, getrandom, BCryptGenRandom, PHP random_bytes, and predictable random generators as contrasts.
What you can do after
Operate/configure secure randomness through standard libraries or suitable operating-system interfaces.

Read how Real-World Cryptography explains entropy →

Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

Entropy is a measure of disorder or randomness in a distribution of class labels.

How it is used
Guides classification-tree split selection by measuring label disorder within candidate subsets.
How deep it goes
Defines disorder and contrasts entropy with information gain and Gini impurity, without giving a formula or worked calculation.
Examples
Uses candidate feature-and-value splits and their resulting target subsets as the practical setting for entropy.
What you can do after
Evaluate candidate classification-tree splits by comparing their ability to reduce label disorder.

Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains entropy →

Grokking AI Algorithms, Second Edition

Entropy is a measure of disorder or uncertainty in a probability distribution or dataset.

How it is used
Guides decision-tree questions toward splits that make class labels less mixed.
How deep it goes
Defines uncertainty and traces split selection, but implements the calculation with Gini terminology rather than a standalone entropy formula.
Examples
Uses diamond quality labels, Carat and Price questions, class-count maps, tree nodes, and Python listings.
What you can do after
Build it from scratch by counting labels, evaluating candidate splits, calculating uncertainty reduction, and constructing tree nodes.

Read how Grokking AI Algorithms, Second Edition explains entropy →

Ensemble Methods for Machine Learning

Entropy is an information-theoretic measure of uncertainty or impurity in a probability distribution.

How it is used
Measures uncertainty for ensemble weighting, decision-tree splitting, feature importance, and neural-network loss functions.
How deep it goes
Defines uncertainty, gives the probability formula, implements it, and applies it to ensembles and decision trees without formal proofs.
Examples
Uses diagnostic certainty, ten classifier labels, entropy-weighted predictions, and an Iris decision tree.
What you can do after
Apply it via NumPy to calculate entropy and via ensemble or decision-tree procedures to weight predictions or select splits.

Read how Ensemble Methods for Machine Learning explains entropy →

Outlier Detection in Python

Entropy is a measure of uncertainty or information content in a probability distribution.

How it is used
Supports categorical outlier detection by combining univariate entropy scores across each row.
How deep it goes
Explains uncertainty, distributions, binning, and implementation while stopping short of modeling feature associations.
Examples
Uses color-frequency distributions, numeric binning, a Python-style detector, and comparative detector outputs.
What you can do after
Apply it by binning numeric features when necessary and scoring categorical rows for univariate outlier detection.

Read how Outlier Detection in Python explains entropy →

Machine Learning Algorithms in Depth

Entropy is a measure of uncertainty in a probability distribution, computed as the expected negative log probability.

How it is used
Serves as an alternative leaf-impurity objective in CART and a monitored uncertainty measure in mean-field image denoising.
How deep it goes
Defines entropy through uncertainty and pure leaves, but works the CART threshold example with Gini rather than entropy.
Examples
Uses a Bernoulli calculation exercise and tracks average binary-pixel entropy during Ising-model image denoising.
What you can do after
Evaluate/understand it as a measure of leaf purity or changing pixel uncertainty, and practice computing it for a Bernoulli variable.

Read how Machine Learning Algorithms in Depth explains entropy →

Applied Reinforcement Learning

Entropy quantifies the uncertainty or randomness of a probability distribution.

How it is used
Uses entropy regularization to encourage exploration during policy-gradient training.
How deep it goes
Defines entropy intuitively, compares Gaussian policies, and gives a regularized objective with a typical coefficient.
Examples
Compares Gaussian policies with mean 0.5 and standard deviations 0.3 and 0.01.
What you can do after
Apply an entropy bonus to encourage exploration while training a policy.

Read how Applied Reinforcement Learning explains entropy →

Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI

A measure of uncertainty in a probability distribution, calculated as the negative sum of each outcome's probability multiplied by its logarithm.

How it is used
Ranks model predictions for uncertainty sampling by how surprising their probability distributions are.
How deep it goes
Defines the negative probability-weighted log sum, shows normalization and base invariance, and explains why equal label probabilities maximize distribution entropy.
Examples
Uses a 50%-chance Detroit Lions game and four-class predictions to illustrate surprise and entropy calculations.
What you can do after
Apply it via PyTorch to calculate normalized entropy scores for probability distributions and rank predictions for sampling.

Read how Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI explains entropy →

Machine Learning System Design

Entropy is a measure of uncertainty or dispersion in a probability distribution.

How it is used
Measures diversity in search results by examining the distribution of result categories.
How deep it goes
Applies entropy as an optional loss component but does not derive its formula or implementation.
Examples
Uses the category distribution of PhotoStock search results as the concrete diversity example.
What you can do after
Evaluate result diversity and consider adding an entropy-based penalty to a search-ranking loss.

Read how Machine Learning System Design explains entropy →

Build a Reasoning Model (From Scratch)

A measure of uncertainty in the model's output distribution during generation, where low entropy indicates collapse and high entropy indicates randomness.

How it is used
Tracks uncertainty during generation to monitor exploration, determinism, and possible training collapse.
How deep it goes
Explains token-level and sequence-average calculations with PyTorch operations, stopping short of broader training analysis.
Examples
Uses seven-token example logits, selected-token log-probabilities, and averaged answer-step entropy.
What you can do after
Apply it via PyTorch to compute token and answer-sequence entropy during model training or generation.

Read how Build a Reasoning Model (From Scratch) explains entropy →

Build AI Drug Discovery Pipelines

Entropy is an uncertainty measure based on the distribution of class probabilities.

How it is used
Evaluates candidate feature splits while constructing classification decision trees from labeled examples.
How deep it goes
Explains entropy’s role in split selection and class proportions, but does not provide its explicit mathematical formula.
Examples
Uses molecular weight, logP, cytochrome P450 inhibition labels, and validation molecules as decision-tree classification examples.
What you can do after
Evaluate feature separations and interpret entropy’s role when training classification decision trees.

Read how Build AI Drug Discovery Pipelines explains entropy →

Also covered in

What to know first

Related concepts

Questions about entropy

What is entropy?

Entropy is an information-theoretic measure of uncertainty in a probability distribution, defined as the expected value of the negative logarithm of outcome probability.

Which Manning books explain entropy?

Grokking Machine Learning, Bayesian Optimization in Action, Timeless Algorithms, The Foundational Ideas, Grokking Machine Learning, Second Edition, Hacking Cryptography, Math and Architectures of Deep Learning, Grokking Bayes, Sutskever's List and 11 more.

What should I know before learning entropy?

Manning books that teach entropy usually introduce accuracy, probability distribution, decision tree and uncertainty first.

What is the difference between entropy and Gini impurity?

Entropy is an information-theoretic measure of uncertainty in a probability distribution, defined as the expected value of the negative logarithm of outcome probability. Gini impurity is a measure of class-label heterogeneity in a set or decision-tree node, calculated from the proportions of its classes.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.