A probability distribution assigns probabilities to possible outcomes. Its probabilities sum to 1, and the distribution determines how predictable the outcomes are. section 2.1.2
Entropy measures the unpredictability of a distribution: a fair coin produces 1 bit, while bias reduces entropy because some outcomes become more likely. section 2.1.2
Equation 2.2
\[\begin{aligned} H(X) &= -\sum_{x \in X} p_x \log_2 p_x \notag\\ &= - p_{1} \times \log_2 (p_{1}) - p_{2} \times \log_2 (p_{2}) - ... p_{n} \times \log_2 (p_{n}) \label{equation:entropy-binary-log} \end{aligned}\]Calculate binary entropy by multiplying each outcome probability by its base-2 logarithm, summing the terms, and negating the result. section 2.1.2
Biased coin
A distribution with P(heads)=0.50 and P(tails)=0.50 has entropy 1.00 bit. Changing it to 0.75 and 0.25 lowers entropy to about 0.81 bits; 0.80 and 0.20 lowers it to 0.72 bits. section 2.1.2
Validate a two-outcome distribution before calculating:
- Accept each probability only between 0 and 1. section 2.1.2
- Require the probabilities to sum to 1, allowing only a small floating-point tolerance. section 2.1.2
