Entropy
Entropy is an information-theoretic measure of uncertainty in a probability distribution, defined as the expected value of the negative logarithm of outcome probability.
Grokking Machine Learning
Entropy is an information-theoretic impurity measure that is higher when class probabilities are more diverse and zero for a pure set.
- How it is used
- Selects classification-tree questions by comparing how mixed their resulting labeled groups are.
- How deep it goes
- Works through split entropies, size-weighted averages, and information gain, without developing information theory beyond tree splitting.
- Examples
- Compares age and platform splits for app recommendations and offers a credit-card fraud exercise.
- What you can do after
- Practice/perform split comparisons using weighted entropy, or apply it via Scikit-Learn’s DecisionTreeClassifier entropy criterion.
Bayesian Optimization in Action
Entropy is an information-theoretic measure of the uncertainty or variability of a random variable, defined for a discrete distribution as the negative sum of each event probability times its base-2 logarithm.
- How it is used
- Uses entropy to choose searches or experiments that most reduce uncertainty about a target quantity.
- How deep it goes
- Defines Bernoulli entropy, develops expected posterior entropy through search examples, and defers closed-form MES scores to sampling approximations.
- Examples
- Uses biased coins, a lost remote, a 100-element sorted list, phone-drop testing, and Forrester-function optimization.
- What you can do after
- Apply it via BoTorch to configure MES policies that approximate expected entropy reduction in Bayesian optimization.
Timeless Algorithms, The Foundational Ideas
Entropy is a quantitative measure of uncertainty across the possible outcomes of an information source and their probabilities.
- How it is used
- Uses entropy to quantify uncertainty in communication and to guide learning decisions.
- How deep it goes
- Defines entropy mathematically and works through bit calculations, compression, and information gain.
- Examples
- Compares fair and biased coins, a die, and a deck of cards, then calculates a decision-tree split.
- What you can do after
- Evaluate and understand source uncertainty and compare candidate decision-tree splits using entropy and information gain.
Read how Timeless Algorithms, The Foundational Ideas explains entropy →
Grokking Machine Learning, Second Edition
A measure of uncertainty or impurity in a probability distribution, commonly used to evaluate classification splits.
- How it is used
- Selects classification-tree questions that make resulting label groups less mixed.
- How deep it goes
- Defines purity through endpoint values and works through weighted split comparisons, without deriving the entropy formula.
- Examples
- Uses app recommendations by age or platform and age-threshold questions with weighted entropy values.
- What you can do after
- Apply it via Scikit-Learn by selecting entropy as a DecisionTreeClassifier criterion and comparing tree splits.
Read how Grokking Machine Learning, Second Edition explains entropy →
Hacking Cryptography
Entropy is a measure of uncertainty in a probability distribution, calculated as the negative sum of each outcome probability multiplied by its logarithmic probability.
- How it is used
- Quantifies RNG unpredictability for choosing and monitoring cryptographic randomness sources.
- How deep it goes
- Defines binary-log entropy, calculates biased-coin values, and applies entropy to TRNG health testing and CSPRNG seeding.
- Examples
- Uses fair and biased coin tosses, including 0.75/0.25 probabilities yielding about 0.81 bits.
- What you can do after
- Evaluate/understand how probability bias lowers unpredictability and why cryptographic systems require monitored quality entropy.
Math and Architectures of Deep Learning
A measure of uncertainty or unpredictability in a probability distribution, high when outcomes are nearly equally likely and low when one outcome dominates.
- How it is used
- Quantifies uncertainty in probability distributions used for Bayesian and unsupervised machine learning.
- How deep it goes
- Defines discrete entropy with worked image calculations, geometric intuition, and Gaussian derivation, then extends it to conditional entropy.
- Examples
- Uses rain forecasts, two four-color images, Gaussian distributions, and predicted cat, dog, airplane, and automobile probabilities.
- What you can do after
- Evaluate distributional uncertainty and compute entropy for discrete distributions or Gaussian distributions with PyTorch.
Read how Math and Architectures of Deep Learning explains entropy →
Grokking Bayes
Entropy is a measure of uncertainty in a random variable, defined as the expected negative logarithm of its probability.
- How it is used
- Measures uncertainty so candidate observations can be compared by their expected reduction in entropy.
- How deep it goes
- Defines discrete entropy, explains its behavior, and works through a numerical calculation; notes differential entropy without developing it.
- Examples
- Uses a uniform prior over 100 integers, a binary-search question, and a Python math.log2 shortcut.
- What you can do after
- Apply it to compare candidate questions by their expected reduction in uncertainty.
Sutskever's List
Entropy is a thermodynamic quantity related to the number of microscopic configurations compatible with a macroscopic state and to the dispersal of energy and matter.
- How it is used
- Frames entropy as the monotonic thermodynamic backdrop for explaining why complexity can rise and fall in a closed system.
- How deep it goes
- Defines entropy through disorder, energy and matter dispersal, and compatible microscopic configurations; illustrates its increase but derives no thermodynamic formulas.
- Examples
- Uses cream mixing in coffee, contrasting layered beginnings, structured swirls, and a uniform endpoint.
- What you can do after
- Understand why rising entropy need not mean rising visible or algorithmic complexity.
Real-World Cryptography
Entropy is a measure of the uncertainty or unpredictability contained in a random source or secret.
- How it is used
- Guides the secure generation of cryptographic secrets and random values.
- How deep it goes
- Defines high and low entropy, then follows randomness through OS seeding, APIs, and early-boot failure modes.
- Examples
- Uses /dev/urandom, getrandom, BCryptGenRandom, PHP random_bytes, and predictable random generators as contrasts.
- What you can do after
- Operate/configure secure randomness through standard libraries or suitable operating-system interfaces.
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
Entropy is a measure of disorder or randomness in a distribution of class labels.
- How it is used
- Guides classification-tree split selection by measuring label disorder within candidate subsets.
- How deep it goes
- Defines disorder and contrasts entropy with information gain and Gini impurity, without giving a formula or worked calculation.
- Examples
- Uses candidate feature-and-value splits and their resulting target subsets as the practical setting for entropy.
- What you can do after
- Evaluate candidate classification-tree splits by comparing their ability to reduce label disorder.
Read how Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI explains entropy →
Grokking AI Algorithms, Second Edition
Entropy is a measure of disorder or uncertainty in a probability distribution or dataset.
- How it is used
- Guides decision-tree questions toward splits that make class labels less mixed.
- How deep it goes
- Defines uncertainty and traces split selection, but implements the calculation with Gini terminology rather than a standalone entropy formula.
- Examples
- Uses diamond quality labels, Carat and Price questions, class-count maps, tree nodes, and Python listings.
- What you can do after
- Build it from scratch by counting labels, evaluating candidate splits, calculating uncertainty reduction, and constructing tree nodes.
Read how Grokking AI Algorithms, Second Edition explains entropy →
Ensemble Methods for Machine Learning
Entropy is an information-theoretic measure of uncertainty or impurity in a probability distribution.
- How it is used
- Measures uncertainty for ensemble weighting, decision-tree splitting, feature importance, and neural-network loss functions.
- How deep it goes
- Defines uncertainty, gives the probability formula, implements it, and applies it to ensembles and decision trees without formal proofs.
- Examples
- Uses diagnostic certainty, ten classifier labels, entropy-weighted predictions, and an Iris decision tree.
- What you can do after
- Apply it via NumPy to calculate entropy and via ensemble or decision-tree procedures to weight predictions or select splits.
Read how Ensemble Methods for Machine Learning explains entropy →
Outlier Detection in Python
Entropy is a measure of uncertainty or information content in a probability distribution.
- How it is used
- Supports categorical outlier detection by combining univariate entropy scores across each row.
- How deep it goes
- Explains uncertainty, distributions, binning, and implementation while stopping short of modeling feature associations.
- Examples
- Uses color-frequency distributions, numeric binning, a Python-style detector, and comparative detector outputs.
- What you can do after
- Apply it by binning numeric features when necessary and scoring categorical rows for univariate outlier detection.
Machine Learning Algorithms in Depth
Entropy is a measure of uncertainty in a probability distribution, computed as the expected negative log probability.
- How it is used
- Serves as an alternative leaf-impurity objective in CART and a monitored uncertainty measure in mean-field image denoising.
- How deep it goes
- Defines entropy through uncertainty and pure leaves, but works the CART threshold example with Gini rather than entropy.
- Examples
- Uses a Bernoulli calculation exercise and tracks average binary-pixel entropy during Ising-model image denoising.
- What you can do after
- Evaluate/understand it as a measure of leaf purity or changing pixel uncertainty, and practice computing it for a Bernoulli variable.
Read how Machine Learning Algorithms in Depth explains entropy →
Applied Reinforcement Learning
Entropy quantifies the uncertainty or randomness of a probability distribution.
- How it is used
- Uses entropy regularization to encourage exploration during policy-gradient training.
- How deep it goes
- Defines entropy intuitively, compares Gaussian policies, and gives a regularized objective with a typical coefficient.
- Examples
- Compares Gaussian policies with mean 0.5 and standard deviations 0.3 and 0.01.
- What you can do after
- Apply an entropy bonus to encourage exploration while training a policy.
Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI
A measure of uncertainty in a probability distribution, calculated as the negative sum of each outcome's probability multiplied by its logarithm.
- How it is used
- Ranks model predictions for uncertainty sampling by how surprising their probability distributions are.
- How deep it goes
- Defines the negative probability-weighted log sum, shows normalization and base invariance, and explains why equal label probabilities maximize distribution entropy.
- Examples
- Uses a 50%-chance Detroit Lions game and four-class predictions to illustrate surprise and entropy calculations.
- What you can do after
- Apply it via PyTorch to calculate normalized entropy scores for probability distributions and rank predictions for sampling.
Machine Learning System Design
Entropy is a measure of uncertainty or dispersion in a probability distribution.
- How it is used
- Measures diversity in search results by examining the distribution of result categories.
- How deep it goes
- Applies entropy as an optional loss component but does not derive its formula or implementation.
- Examples
- Uses the category distribution of PhotoStock search results as the concrete diversity example.
- What you can do after
- Evaluate result diversity and consider adding an entropy-based penalty to a search-ranking loss.
Build a Reasoning Model (From Scratch)
A measure of uncertainty in the model's output distribution during generation, where low entropy indicates collapse and high entropy indicates randomness.
- How it is used
- Tracks uncertainty during generation to monitor exploration, determinism, and possible training collapse.
- How deep it goes
- Explains token-level and sequence-average calculations with PyTorch operations, stopping short of broader training analysis.
- Examples
- Uses seven-token example logits, selected-token log-probabilities, and averaged answer-step entropy.
- What you can do after
- Apply it via PyTorch to compute token and answer-sequence entropy during model training or generation.
Read how Build a Reasoning Model (From Scratch) explains entropy →
Build AI Drug Discovery Pipelines
Entropy is an uncertainty measure based on the distribution of class probabilities.
- How it is used
- Evaluates candidate feature splits while constructing classification decision trees from labeled examples.
- How deep it goes
- Explains entropy’s role in split selection and class proportions, but does not provide its explicit mathematical formula.
- Examples
- Uses molecular weight, logP, cytochrome P450 inhibition labels, and validation molecules as decision-tree classification examples.
- What you can do after
- Evaluate feature separations and interpret entropy’s role when training classification decision trees.
Read how Build AI Drug Discovery Pipelines explains entropy →
Also covered in
- Cyber Threat Hunting
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
- Statistics Every Programmer Needs
- Grokking Web Application Security
- The Art of Code
What to know first
- accuracyAccuracy is the degree to which a result agrees with a specified truth, reference, or criterion, commonly measured as the proportion of evaluated predictions that are correct.
- probability distributionA probability distribution is a mathematical specification assigning probabilities or probability densities to the possible values or outcomes of a random variable.
- decision treeA decision tree is a tree-structured model or representation in which successive tests or choices lead through branches to outcomes, predictions, or actions.
- uncertaintyUncertainty is incomplete knowledge or limited confidence about a value, state, interpretation, behavior, or outcome.
Related concepts
- Gini impurityalternative to · Gini impurity is a measure of class-label heterogeneity in a set or decision-tree node, calculated from the proportions of its classes.
- uncertaintymeasures · Uncertainty is incomplete knowledge or limited confidence about a value, state, interpretation, behavior, or outcome.
- bitmeasured by · A bit is a binary digit and unit of digital information that represents one of two distinguishable values, conventionally 0 or 1.
- costuses · Cost is the monetary expenditure or other resource burden incurred to acquire, produce, operate, or perform something.
- mutual informationcontrasts with · Mutual information is a measure of statistical dependence between random variables, quantifying the information about one variable provided by observing another.
Questions about entropy
What is entropy?
Entropy is an information-theoretic measure of uncertainty in a probability distribution, defined as the expected value of the negative logarithm of outcome probability.
Which Manning books explain entropy?
Grokking Machine Learning, Bayesian Optimization in Action, Timeless Algorithms, The Foundational Ideas, Grokking Machine Learning, Second Edition, Hacking Cryptography, Math and Architectures of Deep Learning, Grokking Bayes, Sutskever's List and 11 more.
What should I know before learning entropy?
Manning books that teach entropy usually introduce accuracy, probability distribution, decision tree and uncertainty first.
What is the difference between entropy and Gini impurity?
Entropy is an information-theoretic measure of uncertainty in a probability distribution, defined as the expected value of the negative logarithm of outcome probability. Gini impurity is a measure of class-label heterogeneity in a set or decision-tree node, calculated from the proportions of its classes.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.


















