Entropy is a measure of disorder or randomness in a distribution of class labels.
Entropy measures how disordered or random the class labels are within a subset. Decision trees use it to prefer splits that produce more homogeneous target groups. section 5.1
In classification-tree training, entropy is related to two other split criteria:
Information gain derives from entropy and measures how much a feature-based split reduces uncertainty about class labels. section 5.1
Gini impurity measures the probability of misclassifying a randomly selected subset element when labels follow the subset’s label distribution. section 5.1
The algorithm searches observed feature values for the combination that creates the most homogeneous target subsets; regression uses different criteria. section 5.1
How Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI covers it
How deep it goes
Defines disorder and contrasts entropy with information gain and Gini impurity, without giving a formula or worked calculation.
Examples
Uses candidate feature-and-value splits and their resulting target subsets as the practical setting for entropy.
What you can do after
Evaluate candidate classification-tree splits by comparing their ability to reduce label disorder.
How it connects
Connects entropy to decision trees, classification, information gain, Gini impurity, feature splits, and homogeneous target subsets.
Grokking Machine LearningEntropy is an information-theoretic impurity measure that is higher when class probabilities are more diverse and zero for a pure set.
Bayesian Optimization in ActionEntropy is an information-theoretic measure of the uncertainty or variability of a random variable, defined for a discrete distribution as the negative sum of each event probability times its base-2 logarithm.
Hacking CryptographyEntropy is a measure of uncertainty in a probability distribution, calculated as the negative sum of each outcome probability multiplied by its logarithmic probability.
Math and Architectures of Deep LearningA measure of uncertainty or unpredictability in a probability distribution, high when outcomes are nearly equally likely and low when one outcome dominates.
Sutskever's ListEntropy is a thermodynamic quantity related to the number of microscopic configurations compatible with a macroscopic state and to the dispersal of energy and matter.
Machine Learning Algorithms in DepthEntropy is a measure of uncertainty in a probability distribution, computed as the expected negative log probability.
Build a Reasoning Model (From Scratch)A measure of uncertainty in the model's output distribution during generation, where low entropy indicates collapse and high entropy indicates randomness.
Gini impurityalternative to · Gini impurity is a measure of class-label heterogeneity in a set or decision-tree node, calculated from the proportions of its classes.
uncertaintymeasures · Uncertainty is incomplete knowledge or confidence about a value, state, interpretation, behavior, or outcome.
bitmeasured by · A bit is a binary digit and unit of digital information that represents one of two distinguishable values, conventionally 0 or 1.
probability distributionmeasures · A probability distribution is a mathematical specification assigning probabilities to the possible values or outcomes of a random variable, with total probability one.
This explainer was generated by Manning's knowledge graph from the text of Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI. The section links point to the parts of the book it draws on.
