chapter five

5 The birth of information theory: Claude Shannon and the mathematics of uncertainty

 

This chapter covers

  • Claude Shannon’s A Mathematical Theory of Communication (1948), which defined information as measurable uncertainty and transformed communication into science
  • His introduction of entropy, measured in bits, as a precise measure of uncertainty and information
  • His intended breakthroughs in communication—capacity, redundancy, coding, and noise—that immediately solved core engineering challenges of that era
  • How those same ideas unexpectedly reshaped statistics, data science, and artificial intelligence through entropy and mutual information
  • How entropy became a foundation of modern algorithms, from decision trees and random forests to clustering, neural networks, and representation learning

5.1 Primers on information and entropy

5.1.1 Information: from meaning to uncertainty

5.1.2 Entropy: quantifying uncertainty

5.2 Shannon’s framework of communication: his intended contribution

5.2.1 Channel capacity

5.2.2 Noisy channel coding

5.2.3 Source coding (compression)

5.2.4 Signal processing and modulation

5.2.5 Why it mattered—and where it led

5.3 Entropy and information gain in data partitioning

5.3.1 Decision trees

5.3.2 Random forests

5.3.3 Feature selection

5.3.4 Clustering

5.4 Entropy and uncertainty reduction in deep learning

5.4.1 Neural networks

5.4.2 Representation learning

5.5 From communication to universal uncertainty

5.6 Summary