Entropy measures uncertainty or impurity in a probability distribution. A disorderly set has higher entropy; a nearly certain outcome has lower entropy. Three diagnoses illustrate the idea: Ana’s 90% healthy result and Bob’s 95% ill result are both relatively certain, while Cam’s 50/50 result is most uncertain. section 3.2.3
For outcome probabilities p, compute information entropy as H(p) = −Σ p log₂(p). The calculation combines the number of possible values with the uncertainty associated with each value. A balanced distribution therefore carries more uncertainty than one concentrated on a single outcome. section 3.2.3
Entropy appears in several machine-learning workflows:
- Decision trees use entropy as a split-quality criterion, choosing splits that reduce weighted entropy and then normalizing the resulting feature importances. section 9.1.2
- Ensemble weighting uses prediction entropy to assess base estimators; because lower entropy is preferred, weights are made inversely proportional to entropy. section 3.2.3
- Neural networks also use entropy-based measures as loss functions. section 3.2.3
Entropy from ensemble predictions
Ten estimators predict [1, 1, 1, 0, 0, 1, 1, 1, 0, 0]. The label probabilities are P(1) = 0.6 and P(0) = 0.4. A direct implementation counts labels, converts counts to probabilities, applies −p log₂(p), and returns the sum. An entropy-weighted ensemble computes each estimator’s validation entropy, assigns inverse-entropy weights, normalizes them, combines predictions, and rounds the result. section 3.2.3
Listing 3.5 Computing entropy
def entropy(y):
_, counts = np.unique(y, return_counts=True) #1
p = np.array(counts.astype('float') / len(y)) #2
ent = -p.T @ np.log2(p) #3
return ent
Inspect the compact NumPy implementation that counts distinct labels, computes their probabilities, and evaluates the entropy formula. section 3.2.3
Listing 9.1 Training and interpreting decision trees
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
iris = load_iris() #1
Xtrn, Xtst, ytrn, ytst = train_test_split(iris.data, iris.target,
test_size=0.15)
from sklearn import tree
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
model = DecisionTreeClassifier( #2
min_samples_leaf=40, criterion='entropy')
model.fit(Xtrn, ytrn) #3
ypred = model.predict(Xtst)
print('Accuracy = {0:4.3}%'.format(accuracy_score(ytst, ypred) * 100))
import graphviz, re, pydotplus
dot = tree.export_graphviz(model, feature_names=iris.feature_names,
class_names=['Iris-Setosa',
'Iris-Versicolour',
'Iris-Virginica'],
filled=True, impurity=False)
graphviz.Source(dot, format="png") #4
Compare entropy’s from-scratch meaning with its production use as the criterion in a scikit-learn decision-tree classifier. section 9.1.2
