Entropy measures uncertainty in a probability distribution. It is low when one outcome is overwhelmingly likely and high when several outcomes have comparable probabilities. In machine learning, this gives a way to describe whether a distribution is concentrated around a few likely inputs or spread diffusely across many possibilities. section 6.2 section 6.10
For discrete outcomes xᵢ with probabilities p(xᵢ), entropy is H(X) = −Σᵢ p(xᵢ) log p(xᵢ). Each outcome contributes according to both its probability and its surprise: likely outcomes contribute little, while uncertainty rises when probability mass is shared more evenly. Probabilities must sum to 1, so making some outcomes more likely necessarily makes others less likely. section 6.2 section 6.2.1
Colors, prediction, and compression
Consider two 4 × 4 images using green, red, blue, and yellow pixels. When every color occurs four times, each has probability 0.25 and entropy is 1.386294. When green occurs 12 times and the other colors are rare, entropy falls to 0.822265. The first image is harder to predict pixel by pixel; the second can assign green a shorter code and compress more effectively. section 6.2 section 6.2
Table 6.3 Entropy computation for the pair of images in figure 6.1. The right-hand image has lower entropy and can be compressed more.
| Left image |
Right image |
|---|---|
| x1 = G, p(x1) = 4/16 = 0.25 |
x1 = G, p(x1) = 12/16 = 0.75 |
| x2 = R , p(x2) = 4/16 = 0.25 |
x2 = R , p(x2) = 2/16 = 0.125 |
| x3 = B, p(x3) = 4/16 = 0.25 |
x3 = B, p(x3) = 1/16 = 0.0625 |
| x4 = Y, p(x4) = 4/16 = 0.25 |
x4 = Y, p(x4) = 1/16 = 0.0625 |
| ℍ = −(0.25 log(0.25)+0.25 log(0.25) + |
ℍ = −(0.75 log(0.75)+0.125 log(0.125) + |
Compare the probability terms and entropy totals for equally distributed versus green-dominated pixels. section 6.2
The same distinction has geometric form. A flat, wide probability density function corresponds to a diffuse sample cloud and high entropy. Tall, narrow peaks correspond to dense regions of samples and low entropy. For a Gaussian, increasing its spread increases entropy; the univariate result can be computed directly or checked with PyTorch’s distribution entropy method. section 6.2.1 section 6.2.2
Entropy also appears as a component of broader probabilistic objectives:
- Conditional entropy measures the uncertainty remaining in one random variable after another is known, averaging the entropy for each condition by that condition’s probability. section 6.5
- Cross-entropy compares a ground-truth distribution with model predictions and supplies a classification loss: predictions close to the one-hot target receive lower loss. section 6.3
- In a variational autoencoder, entropy is a term in the ELBO; maximizing it encourages a diffuse approximate latent distribution and smoother latent space. section 14.7.6
