Entropy
Entropy is a quantitative measure of uncertainty or dispersion in a probability distribution and, more generally, of disorder in the possible states of a system.
Grokking Machine Learning
Entropy is an information-theoretic impurity measure that is higher when class probabilities are more diverse and zero for a pure set.
- How it is used
- Selects classification-tree questions by comparing how mixed their resulting labeled groups are.
- How deep it goes
- Works through split entropies, size-weighted averages, and information gain, without developing information theory beyond tree splitting.
- Examples
- Compares age and platform splits for app recommendations and offers a credit-card fraud exercise.
- What you can do after
- Practice/perform split comparisons using weighted entropy, or apply it via Scikit-Learn’s DecisionTreeClassifier entropy criterion.
Bayesian Optimization in Action
Entropy is an information-theoretic measure of the uncertainty or variability of a random variable, defined for a discrete distribution as the negative sum of each event probability times its base-2 logarithm.
- How it is used
- Uses entropy to choose searches or experiments that most reduce uncertainty about a target quantity.
- How deep it goes
- Defines Bernoulli entropy, develops expected posterior entropy through search examples, and defers closed-form MES scores to sampling approximations.
- Examples
- Uses biased coins, a lost remote, a 100-element sorted list, phone-drop testing, and Forrester-function optimization.
- What you can do after
- Apply it via BoTorch to configure MES policies that approximate expected entropy reduction in Bayesian optimization.
Hacking Cryptography
Entropy is a measure of uncertainty in a probability distribution, calculated as the negative sum of each outcome probability multiplied by its logarithmic probability.
- How it is used
- Quantifies RNG unpredictability for choosing and monitoring cryptographic randomness sources.
- How deep it goes
- Defines binary-log entropy, calculates biased-coin values, and applies entropy to TRNG health testing and CSPRNG seeding.
- Examples
- Uses fair and biased coin tosses, including 0.75/0.25 probabilities yielding about 0.81 bits.
- What you can do after
- Evaluate/understand how probability bias lowers unpredictability and why cryptographic systems require monitored quality entropy.
Math and Architectures of Deep Learning
A measure of uncertainty or unpredictability in a probability distribution, high when outcomes are nearly equally likely and low when one outcome dominates.
- How it is used
- Quantifies uncertainty in probability distributions used for Bayesian and unsupervised machine learning.
- How deep it goes
- Defines discrete entropy with worked image calculations, geometric intuition, and Gaussian derivation, then extends it to conditional entropy.
- Examples
- Uses rain forecasts, two four-color images, Gaussian distributions, and predicted cat, dog, airplane, and automobile probabilities.
- What you can do after
- Evaluate distributional uncertainty and compute entropy for discrete distributions or Gaussian distributions with PyTorch.
Read how Math and Architectures of Deep Learning explains entropy →
Real-World Cryptography
Entropy is a measure of the uncertainty or unpredictability contained in a random source or secret.
- How it is used
- Guides the secure generation of cryptographic secrets and random values.
- How deep it goes
- Defines high and low entropy, then follows randomness through OS seeding, APIs, and early-boot failure modes.
- Examples
- Uses /dev/urandom, getrandom, BCryptGenRandom, PHP random_bytes, and predictable random generators as contrasts.
- What you can do after
- Operate/configure secure randomness through standard libraries or suitable operating-system interfaces.
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
Entropy is a measure of disorder or randomness in a distribution of class labels.
- How it is used
- Guides classification-tree split selection by measuring label disorder within candidate subsets.
- How deep it goes
- Defines disorder and contrasts entropy with information gain and Gini impurity, without giving a formula or worked calculation.
- Examples
- Uses candidate feature-and-value splits and their resulting target subsets as the practical setting for entropy.
- What you can do after
- Evaluate candidate classification-tree splits by comparing their ability to reduce label disorder.
Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains entropy →
Grokking AI Algorithms, Second Edition
Entropy is a measure of disorder or uncertainty in a probability distribution or dataset.
- How it is used
- Guides decision-tree questions toward splits that make class labels less mixed.
- How deep it goes
- Defines uncertainty and traces split selection, but implements the calculation with Gini terminology rather than a standalone entropy formula.
- Examples
- Uses diamond quality labels, Carat and Price questions, class-count maps, tree nodes, and Python listings.
- What you can do after
- Build it from scratch by counting labels, evaluating candidate splits, calculating uncertainty reduction, and constructing tree nodes.
Read how Grokking AI Algorithms, Second Edition explains entropy →
Sutskever's List
Entropy is a thermodynamic quantity related to the number of microscopic configurations compatible with a macroscopic state and to the dispersal of energy and matter.
- How it is used
- Frames entropy as the monotonic thermodynamic backdrop for explaining why complexity can rise and fall in a closed system.
- How deep it goes
- Defines entropy through disorder, energy and matter dispersal, and compatible microscopic configurations; illustrates its increase but derives no thermodynamic formulas.
- Examples
- Uses cream mixing in coffee, contrasting layered beginnings, structured swirls, and a uniform endpoint.
- What you can do after
- Understand why rising entropy need not mean rising visible or algorithmic complexity.
Ensemble Methods for Machine Learning
Entropy is an information-theoretic measure of uncertainty or impurity in a probability distribution.
- How it is used
- Measures uncertainty for ensemble weighting, decision-tree splitting, feature importance, and neural-network loss functions.
- How deep it goes
- Defines uncertainty, gives the probability formula, implements it, and applies it to ensembles and decision trees without formal proofs.
- Examples
- Uses diagnostic certainty, ten classifier labels, entropy-weighted predictions, and an Iris decision tree.
- What you can do after
- Apply it via NumPy to calculate entropy and via ensemble or decision-tree procedures to weight predictions or select splits.
Read how Ensemble Methods for Machine Learning explains entropy →
Outlier Detection in Python
Entropy is a measure of uncertainty or information content in a probability distribution.
- How it is used
- Supports categorical outlier detection by combining univariate entropy scores across each row.
- How deep it goes
- Explains uncertainty, distributions, binning, and implementation while stopping short of modeling feature associations.
- Examples
- Uses color-frequency distributions, numeric binning, a Python-style detector, and comparative detector outputs.
- What you can do after
- Apply it by binning numeric features when necessary and scoring categorical rows for univariate outlier detection.
Machine Learning Algorithms in Depth
Entropy is a measure of uncertainty in a probability distribution, computed as the expected negative log probability.
- How it is used
- Serves as an alternative leaf-impurity objective in CART and a monitored uncertainty measure in mean-field image denoising.
- How deep it goes
- Defines entropy through uncertainty and pure leaves, but works the CART threshold example with Gini rather than entropy.
- Examples
- Uses a Bernoulli calculation exercise and tracks average binary-pixel entropy during Ising-model image denoising.
- What you can do after
- Evaluate/understand it as a measure of leaf purity or changing pixel uncertainty, and practice computing it for a Bernoulli variable.
Read how Machine Learning Algorithms in Depth explains entropy →
Machine Learning System Design
Entropy is a measure of uncertainty or dispersion in a probability distribution.
- How it is used
- Measures diversity in search results by examining the distribution of result categories.
- How deep it goes
- Applies entropy as an optional loss component but does not derive its formula or implementation.
- Examples
- Uses the category distribution of PhotoStock search results as the concrete diversity example.
- What you can do after
- Evaluate result diversity and consider adding an entropy-based penalty to a search-ranking loss.
Build a Reasoning Model (From Scratch)
A measure of uncertainty in the model's output distribution during generation, where low entropy indicates collapse and high entropy indicates randomness.
- How it is used
- Tracks uncertainty during generation to monitor exploration, determinism, and possible training collapse.
- How deep it goes
- Explains token-level and sequence-average calculations with PyTorch operations, stopping short of broader training analysis.
- Examples
- Uses seven-token example logits, selected-token log-probabilities, and averaged answer-step entropy.
- What you can do after
- Apply it via PyTorch to compute token and answer-sequence entropy during model training or generation.
Read how Build a Reasoning Model (From Scratch) explains entropy →
Related concepts
- Gini impurityalternative to · Gini impurity is a measure of class-label heterogeneity in a set or decision-tree node, calculated from the proportions of its classes.
- uncertaintymeasures · Uncertainty is incomplete knowledge or confidence about a value, state, interpretation, behavior, or outcome.
- bitmeasured by · A bit is a binary digit and unit of digital information that represents one of two distinguishable values, conventionally 0 or 1.
- probability distributionmeasures · A probability distribution is a mathematical specification assigning probabilities to the possible values or outcomes of a random variable, with total probability one.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.