Entropy measures uncertainty in a probability distribution. For class probabilities πₖ, it is the expected negative log probability: H = −Σₖ πₖ log πₖ. A pure decision-tree leaf has one class with probability 1 and entropy 0; a leaf with uncertain class membership has higher entropy. section 5.6
In a CART decision tree, entropy is an alternative to the Gini index for judging how well a split separates classes. The aim is to reduce leaf impurity while limiting tree depth to guard against overfitting. The worked threshold calculation uses Gini, not entropy. section 5.6
Entropy during image denoising
Mean-field variational inference tracks the average entropy of binary pixel distributions alongside its ELBO objective. In the reported Ising-model denoising run, average pixel entropy decreases over iterations while ELBO increases. section 3.3
For practice, compute H(X) for a Bernoulli random variable, alongside its expectation and variance. section 3.5
