chapter five

5 Entropy and the bit: Claude Shannon and the quantification of information

This chapter covers

Claude Shannon’s A Mathematical Theory of Communication (1948), which transformed communication into a mathematical science
Why entropy became a mathematical measure of uncertainty and information
How the bit became the unit for measuring entropy
How Shannon’s redefinition of information solved the engineering challenges of communication
How those same mathematical ideas were later repurposed to power modern learning methods

So far, uncertainty has been examined through the work of some of its greatest interpreters. Bayes showed how beliefs could be updated as new evidence arrived, reducing uncertainty by replacing prior beliefs with more informed posterior ones. Fisher developed maximum likelihood estimation, reducing uncertainty about unknown parameters by identifying the values most consistent with the observed data. Neyman and Pearson established a framework for hypothesis testing, reducing uncertainty by providing an objective basis for drawing conclusions and taking action while controlling the risks of statistical error. Each advanced the science of reasoning under uncertainty—but none provided a way to measure uncertainty itself.

5.1 How Shannon redefined information

5.1.1 Information as the reduction of uncertainty

5.1.2 Entropy as the measure of uncertainty

5.1.3 Measuring entropy in bits

5.1.4 Entropy, redundancy, and compression

5.2 Shannon’s framework of communication

5.2.1 Source coding (compression)

5.2.2 Channel capacity

5.2.3 Noisy channel coding

5.2.4 Signal design

5.2.5 From communication to modern learning methods

5.3 From information theory to data partitioning

5.3.1 Decision trees

5.3.2 Random forests

5.3.3 Feature selection

5.3.4 Clustering

5.4 Information theory in deep learning

5.4.1 Neural networks

5.4.2 Representation learning

5.5 From communication to learning

5.6 Summary