liveBook · Manning

Chapter 5: Entropy and the bit: Claude Shannon and the quantification of information

What you will learn in chapter 5 of Timeless Algorithms, The Foundational Ideas

The introduction contrasts earlier ways of reducing uncertainty with Shannon’s way of measuring it, then shows why that shift mattered for communication.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction contrasts earlier ways of reducing uncertainty with Shannon’s way of measuring it, then shows why that shift mattered for communication.

Key terms: information, uncertainty, redundancy, efficiency, reliability

5.1 How Shannon redefined information

Shannon reframes information as a reduction in uncertainty rather than a message’s meaning or length.

Key terms: information, uncertainty, message

5.1.1 Information as the reduction of uncertainty

This section explains information as uncertainty resolved by an observed outcome and shows how redundancy affects compression and reliable transmission.

Key terms: information, reduction of uncertainty, possible outcomes, probabilities, redundancy, compression, transmission errors, noise

5.1.2 Entropy as the measure of uncertainty

This section defines entropy as a source’s average uncertainty, traces its adaptation from physical entropy, and explains how outcome probabilities affect it.

Key terms: entropy, information source, average uncertainty, observed outcome, Boltzmann, Gibbs, Shannon, microstates, 𝐻(𝑋), certain outcome

5.1.3 Measuring entropy in bits

This section uses bits and worked examples to show how Shannon’s entropy equation puts different sources of uncertainty on a common scale.

Key terms: bit, binary digit, entropy, fair coin, biased coin, entropy of the source, information conveyed, probability-weighted average, equally likely outcomes, Shannon's entropy equation

5.1.4 Entropy, redundancy, and compression

This section connects entropy to how much a message can be compressed and explains why some redundancy is removed while other redundancy is added to protect transmission.

Key terms: entropy, bits, information source, low-entropy sources, high-entropy sources, redundancy, compression, channel capacity, noise, transmission errors

5.2 Shannon’s framework of communication

This section introduces Shannon’s common framework for communication, tracing a message from source to receiver and showing how uncertainty, channel limits, and redundancy can be analyzed together.

Key terms: information source, encoder, channel, noise, decoder, receiver, source entropy, channel capacity, compression, redundancy

5.2.1 Source coding (compression)

This section explains how source coding removes redundancy and how entropy sets the limit for lossless compression.

Key terms: encoder, source coding, redundancy, Shannon's source coding theorem, entropy, lossless compression, lossless, lossy, ZIP, PNG

5.2.2 Channel capacity

The section defines channel capacity and explains why reliable communication requires a compressed message to fit the channel’s limit.

Key terms: channel capacity, bandwidth, noise, source coding theorem, lossless limit, lossy compression

5.2.3 Noisy channel coding

The section explains how deliberately added redundancy can protect messages from noise, and states the capacity condition under which coding can make errors arbitrarily rare.

Key terms: noise, channel coding, structured redundancy, decoder, error-correcting codes, majority voting, noisy channel coding theorem, transmission rate, channel capacity, probability of error

5.2.4 Signal design

Signal design turns a compressed, error-protected message into physical signals while balancing how much information they carry against how reliably they can be distinguished in noise.

Key terms: physical signals, communication channel, digital modulation, distinguishable signal states, noise, channel's capacity

5.2.5 From communication to modern learning methods

The section explains why Shannon’s measurable view of information extends from communication systems to modern learning methods.

Key terms: information, uncertainty, entropy, communication

5.3 From information theory to data partitioning

This introduction explains how Shannon’s measure of information extends from communication to learning from data.

Key terms: information, reduction of uncertainty, entropy, bits, observation

5.3.1 Decision trees

This section explains how decision trees use entropy and information gain to choose splits, illustrated with a loan example, and how those splits relate to variable importance.

Key terms: root node, internal node, branch, leaf node, classification, regression, entropy, information gain, split, homogeneous groups

5.3.2 Random forests

This section explains how random forests combine diverse decision trees while using entropy reduction to choose splits and assess predictor importance.

Key terms: overfit, bootstrapped sample, random subset of predictors, majority voting, averaging, entropy, uncertainty, information gain, candidate split, variable importance

5.3.3 Feature selection

This section shows how mutual information measures a predictor’s reduction of uncertainty about a target and guides feature selection.

Key terms: feature selection, predictors, target variable, mutual information, entropy, conditional entropy, bits, information gain, decision trees

5.3.4 Clustering

The section explains how entropy complements geometric clustering by measuring how much cluster membership reduces uncertainty about meaningful categories, and how information can also guide cluster formation.

Key terms: clustering, unlabeled data, k-means clustering, centroid, hierarchical clustering, dendrogram, geometry, feature space, entropy-based evaluation, cluster membership

5.4 Information theory in deep learning

This section shows how Shannon’s ideas about information, uncertainty, and redundancy apply to neural-network training and representation learning.

Key terms: cross-entropy loss function, predicted probabilities, maximum likelihood estimation, representation learning, high-dimensional, redundancy, noise, downstream tasks

5.4.1 Neural networks

This section explains how cross-entropy turns a neural network’s probabilistic predictions into a measurable training objective, using digit recognition to illustrate uncertainty reduction.

Key terms: cross-entropy loss, predicted distribution, true distribution of outcomes, weights and biases, parameters, optimization, Shannon's entropy, bits, digit-classification network

5.4.2 Representation learning

This section explains representation learning as Shannon-inspired compression that removes redundancy while preserving information useful for a task.

Key terms: representation learning, representations, raw data, source coding, entropy, redundancy, informative content, encoder, latent representation, decoder

5.5 From communication to learning

This section summarizes how Shannon’s measurement of uncertainty gave communication engineering a mathematical foundation and became useful for learning from data.

Key terms: information, uncertainty, entropy, redundancy, bits, compression, transmission, reliable communication, bandwidth constraints, noise

5.6 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 4: From evidence to decisions: Neyman, Pearson, and the principles of hypothesis testing Chapter 6: Dynamic programming: Richard Bellman and recursive optimization →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.