Chapter 5: Entropy and the bit: Claude Shannon and the quantification of information
The introduction contrasts earlier ways of reducing uncertainty with Shannon’s way of measuring it, then shows why that shift mattered for communication.
After this chapter you will be able to
- Information as resolved uncertainty. Explain how Shannon’s view differed from the earlier approaches described here: Bayes, Fisher, and Neyman and Pearson reduced uncertainty in different ways, while Shannon made uncertainty itself measurable by treating information as the uncertainty it resolves.
- Redundancy and communication goals. Describe the two roles of redundancy identified in the introduction: removing unnecessary redundancy makes transmission more efficient, while carefully designed redundancy protects a message when reliability is at risk.
- Information as uncertainty reduction. Explain why, for accurate and efficient transmission, Shannon treats information as a reduction in uncertainty rather than as a message’s meaning or amount of content. Contrast a long expected message with a short message that resolves a highly uncertain event.
- Information as resolved uncertainty. Explain why the information conveyed by an outcome depends on how uncertain it was beforehand, rather than on the message’s length or meaning. Describe how both the number and the probabilities of possible outcomes affect that uncertainty.
- Redundancy in communication. Distinguish naturally occurring redundancy that permits compression from deliberately added redundancy that helps protect a message against transmission errors. Use the contrasting example sequences to explain why a repeated pattern is easier to compress than an unpredictable one.
- Source uncertainty versus outcome information. Distinguish the information conveyed by one observed outcome from the entropy of an entire source before an outcome is known.
- Probability-based entropy framework. Trace how Shannon adapted entropy from physics into a measure for an information source, and describe the role of its possible outcomes and their probabilities.
- The bit as a unit. Explain what one bit represents and why a fair coin has an entropy of one bit.
- Source entropy versus outcome information. Distinguish the entropy of a biased coin from the information conveyed by observing one of its outcomes.
- Entropy across outcome sets. Describe how the entropy calculation and its result change as the number of equally likely outcomes grows from a fair coin to a die to a deck of cards.
- Predictability and compression potential. Explain why a predictable sequence such as AAAAAA can be represented compactly, while an unpredictable sequence such as W7dX8$ offers little redundancy to remove.
- Two purposes of redundancy. Distinguish naturally occurring redundancy that can be removed from redundancy deliberately added before transmission to help detect and often correct noise-induced errors.
- Communication stages and roles. Trace a message from an information source through an encoder and channel to a decoder and receiver, describing what each stage does and where noise can affect the signal.
- Uncertainty capacity and redundancy. Explain how source entropy and channel capacity pose different communication questions, and distinguish removing redundancy for compression from adding structured redundancy for protection against noise.
- Redundancy removal in source coding. Explain how an encoder uses source coding to represent a message with fewer bits before transmission while preserving its information.
- Entropy limit on lossless compression. Explain what Shannon’s source coding theorem says entropy determines, using the section’s 1,000-bit example to distinguish removable redundancy from information that cannot be compressed away.
- Meaning of channel capacity. Explain channel capacity as the maximum rate at which information can be transmitted while the original message remains recoverable with an arbitrarily small probability of error. Describe how bandwidth and noise affect that limit.
- Matching source and channel limits. Explain why the information remaining after lossless compression must fit within the channel’s capacity for reliable transmission. Apply the section’s 700-bit message and 600-bit interval example to describe the mismatch and the available options.
- Structured redundancy for error correction. Explain how channel coding adds structured redundancy so a decoder can detect and sometimes correct changes caused by noise. Describe how repeating the letters of YES allows majority voting to recover a corrupted symbol.
- Capacity condition for reliability. State Shannon’s noisy channel coding result: when the transmission rate remains below channel capacity, coding schemes can make the probability of error arbitrarily small. Explain why fitting a message within capacity alone does not prevent noise from corrupting it.
- Signal efficiency and reliability tradeoff. Explain how the number of distinguishable signal states affects both the information conveyed and susceptibility to noise, and why signal design balances the two.
- Measurable Information Beyond Communication. Explain how defining information in terms of uncertainty made it useful beyond communication, and contrast the aims of communication systems with those of learning methods.
- Information extracted from data. Explain how treating information as a measurable reduction in uncertainty lets learning algorithms use Shannon’s idea to learn from observations, not just analyze transmitted messages.
- Entropy guided split selection. Explain how entropy and information gain guide the choice among candidate splits and why the procedure repeats down the tree.
- Loan split information gain. Trace the loan example’s comparison of uncertainty before and after its proposed split, including how branch entropies are combined.
- Random Forest Construction and Prediction. Explain why a single deep decision tree may overfit and how a random forest addresses that problem. Describe the two sources of variation among its trees and how their predictions are combined.
- Entropy Guided Forest Splits. Describe how entropy and information gain guide the splits inside a random forest’s individual trees.
- Mutual information calculation. Describe mutual information as the difference between a target’s entropy before a predictor is known and the uncertainty remaining after it is known. Trace how the patient example uses the blood-pressure groups to obtain that difference.
- Predictor ranking versus tree splits. Distinguish using entropy reduction to evaluate a predictor for feature selection from using it to choose a decision-tree split, and apply the distinction to blood pressure and shoe size.
- Geometric versus informational evaluation. Distinguish geometric cluster evaluation from entropy-based evaluation, including when known categories are needed for the latter comparison.
- Category agreement and cluster measures. Explain what cluster entropy, purity, and normalized mutual information indicate, and use the fruit example to compare pure and mixed clusters.
- Cross-entropy in neural training. Explain that cross-entropy measures how closely a neural network’s predicted probabilities match true outcomes, and that training adjusts parameters to reduce this loss. State the section’s connection between minimizing cross-entropy and maximizing the likelihood of observed data.
- Compact informative representations. Explain why learning directly from high-dimensional, redundant raw data can be inefficient, and describe what representation learning aims to retain and remove.
- Cross-entropy as prediction measure. Explain how cross-entropy measures the discrepancy between a network’s predicted probabilities and the true outcomes, and why evaluating the whole distribution is more informative than checking whether the selected class is right.
- Incremental loss-guided learning. Trace how a network uses cross-entropy loss during training: after predictions, it makes small adjustments to weights and biases according to their contribution to the error, repeatedly shifting probability toward correct outcomes.
- Purpose of learned representations. Explain that representation learning transforms raw, high-dimensional data into compact, structured descriptions useful for later tasks, rather than merely producing predictions. Distinguish the broader idea from neural networks, which commonly implement it.
- Information preserving compression. Explain how the section connects representation learning to Shannon’s source coding and entropy: predictable variation contributes little new information, but compression is valuable only if it retains information that matters.
- Information and entropy in bits. Explain that Shannon defined information as a reduction of uncertainty and entropy as a measure of average uncertainty, making information, uncertainty, and redundancy quantifiable in bits.
- Information measures in learning. Explain why Shannon’s ideas extend beyond communication: once information and uncertainty can be measured, they can be optimized in methods for learning from data.
What it builds on
- From prior to posterior: how Bayes’ Theorem works (chapter 2) · Earlier treatment of updating beliefs with evidence.
- Maximum likelihood estimation (chapter 3) · Earlier treatment of estimating unknown parameters.
- The stepwise framework of hypothesis testing (chapter 4) · Earlier treatment of drawing conclusions while controlling statistical errors.
Section by section
Introduction
The introduction contrasts earlier ways of reducing uncertainty with Shannon’s way of measuring it, then shows why that shift mattered for communication.
- Information as resolved uncertainty: Explain how Shannon’s view differed from the earlier approaches described here: Bayes, Fisher, and Neyman and Pearson reduced uncertainty in different ways, while Shannon made uncertainty itself measurable by treating information as the uncertainty it resolves.
Why it matters: This distinction is the starting point for the chapter’s treatment of information as something measurable. - Redundancy and communication goals: Describe the two roles of redundancy identified in the introduction: removing unnecessary redundancy makes transmission more efficient, while carefully designed redundancy protects a message when reliability is at risk.
Why it matters: It explains why efficient communication and reliable communication may call for different uses of redundancy.
Key terms: information, uncertainty, redundancy, efficiency, reliability
5.1 How Shannon redefined information
Shannon reframes information as a reduction in uncertainty rather than a message’s meaning or length.
- Information as uncertainty reduction: Explain why, for accurate and efficient transmission, Shannon treats information as a reduction in uncertainty rather than as a message’s meaning or amount of content. Contrast a long expected message with a short message that resolves a highly uncertain event.
Why it matters: This distinction explains why message length alone cannot tell you how informative a message is.
Key terms: information, uncertainty, message
5.1.1 Information as the reduction of uncertainty
This section explains information as uncertainty resolved by an observed outcome and shows how redundancy affects compression and reliable transmission.
- Information as resolved uncertainty: Explain why the information conveyed by an outcome depends on how uncertain it was beforehand, rather than on the message’s length or meaning. Describe how both the number and the probabilities of possible outcomes affect that uncertainty.
Why it matters: This is the intuition behind Shannon’s way of quantifying information. - Redundancy in communication: Distinguish naturally occurring redundancy that permits compression from deliberately added redundancy that helps protect a message against transmission errors. Use the contrasting example sequences to explain why a repeated pattern is easier to compress than an unpredictable one.
Why it matters: The two uses of redundancy connect Shannon’s account of information to both efficient and reliable communication.
Key terms: information, reduction of uncertainty, possible outcomes, probabilities, redundancy, compression, transmission errors, noise
5.1.2 Entropy as the measure of uncertainty
This section defines entropy as a source’s average uncertainty, traces its adaptation from physical entropy, and explains how outcome probabilities affect it.
- Source uncertainty versus outcome information: Distinguish the information conveyed by one observed outcome from the entropy of an entire source before an outcome is known.
Why it matters: The distinction keeps a surprising result from being confused with the uncertainty of the source that produced it. - Probability-based entropy framework: Trace how Shannon adapted entropy from physics into a measure for an information source, and describe the role of its possible outcomes and their probabilities.
Why it matters: This explains why entropy can assign a numerical uncertainty to a source without treating information as a physical quantity. - Entropy at probability extremes: Explain when a source has zero or maximum entropy and relate predictability to uncertainty.
Why it matters: These extremes give you a way to check whether a source is highly predictable or highly uncertain.
Key terms: entropy, information source, average uncertainty, observed outcome, Boltzmann, Gibbs, Shannon, microstates, 𝐻(𝑋), certain outcome
5.1.3 Measuring entropy in bits
This section uses bits and worked examples to show how Shannon’s entropy equation puts different sources of uncertainty on a common scale.
- The bit as a unit: Explain what one bit represents and why a fair coin has an entropy of one bit.
Why it matters: The bit gives the other entropy values in the section a common unit. - Source entropy versus outcome information: Distinguish the entropy of a biased coin from the information conveyed by observing one of its outcomes.
Why it matters: It prevents confusion between a source’s average uncertainty and the surprise of one result. - Entropy across outcome sets: Describe how the entropy calculation and its result change as the number of equally likely outcomes grows from a fair coin to a die to a deck of cards.
Why it matters: Bits make the uncertainty of very different sources directly comparable.
Key terms: bit, binary digit, entropy, fair coin, biased coin, entropy of the source, information conveyed, probability-weighted average, equally likely outcomes, Shannon's entropy equation
5.1.4 Entropy, redundancy, and compression
This section connects entropy to how much a message can be compressed and explains why some redundancy is removed while other redundancy is added to protect transmission.
- Entropy as comparable uncertainty: Explain that entropy measures uncertainty in bits and depends on the possible outcomes and their probabilities, allowing different information sources to be compared.
Why it matters: A common measure of uncertainty makes the compression comparison meaningful. - Predictability and compression potential: Explain why a predictable sequence such as AAAAAA can be represented compactly, while an unpredictable sequence such as W7dX8$ offers little redundancy to remove.
Why it matters: A source’s predictability indicates how many bits compression might save. - Two purposes of redundancy: Distinguish naturally occurring redundancy that can be removed from redundancy deliberately added before transmission to help detect and often correct noise-induced errors.
Why it matters: Efficient communication may require removing some redundancy while adding other redundancy for reliability.
Key terms: entropy, bits, information source, low-entropy sources, high-entropy sources, redundancy, compression, channel capacity, noise, transmission errors
5.2 Shannon’s framework of communication
This section introduces Shannon’s common framework for communication, tracing a message from source to receiver and showing how uncertainty, channel limits, and redundancy can be analyzed together.
- Communication stages and roles: Trace a message from an information source through an encoder and channel to a decoder and receiver, describing what each stage does and where noise can affect the signal.
Why it matters: The stages provide a common way to describe what happens to a message in different communication technologies. - Uncertainty capacity and redundancy: Explain how source entropy and channel capacity pose different communication questions, and distinguish removing redundancy for compression from adding structured redundancy for protection against noise.
Why it matters: It separates the problem of using fewer bits from the problem of recovering information despite noise. - Unified mathematical communication framework: Explain what Shannon unified and why that changed how communication problems could be addressed.
Why it matters: The framework lets seemingly different communication technologies be examined using the same questions and measures.
Key terms: information source, encoder, channel, noise, decoder, receiver, source entropy, channel capacity, compression, redundancy
5.2.1 Source coding (compression)
This section explains how source coding removes redundancy and how entropy sets the limit for lossless compression.
- Redundancy removal in source coding: Explain how an encoder uses source coding to represent a message with fewer bits before transmission while preserving its information.
Why it matters: It explains why a shorter transmitted representation can still preserve the whole message. - Entropy limit on lossless compression: Explain what Shannon’s source coding theorem says entropy determines, using the section’s 1,000-bit example to distinguish removable redundancy from information that cannot be compressed away.
Why it matters: Entropy tells us the limit a lossless encoder faces, regardless of its design. - Lossless and lossy formats: Distinguish lossless formats from lossy formats by what each preserves or discards to achieve compression.
Why it matters: The distinction clarifies why greater compression may come at the cost of exact recovery.
Key terms: encoder, source coding, redundancy, Shannon's source coding theorem, entropy, lossless compression, lossless, lossy, ZIP, PNG
5.2.2 Channel capacity
The section defines channel capacity and explains why reliable communication requires a compressed message to fit the channel’s limit.
- Meaning of channel capacity: Explain channel capacity as the maximum rate at which information can be transmitted while the original message remains recoverable with an arbitrarily small probability of error. Describe how bandwidth and noise affect that limit.
Why it matters: Compression alone does not guarantee that a message can be delivered reliably through a particular channel. - Matching source and channel limits: Explain why the information remaining after lossless compression must fit within the channel’s capacity for reliable transmission. Apply the section’s 700-bit message and 600-bit interval example to describe the mismatch and the available options.
Why it matters: It explains why engineers must design compression and transmission around the channel’s limit rather than assuming compression can always make a message fit.
Key terms: channel capacity, bandwidth, noise, source coding theorem, lossless limit, lossy compression
5.2.3 Noisy channel coding
The section explains how deliberately added redundancy can protect messages from noise, and states the capacity condition under which coding can make errors arbitrarily rare.
- Structured redundancy for error correction: Explain how channel coding adds structured redundancy so a decoder can detect and sometimes correct changes caused by noise. Describe how repeating the letters of YES allows majority voting to recover a corrupted symbol.
Why it matters: Reliable communication can come from making a message resilient to noise, rather than eliminating noise. - Capacity condition for reliability: State Shannon’s noisy channel coding result: when the transmission rate remains below channel capacity, coding schemes can make the probability of error arbitrarily small. Explain why fitting a message within capacity alone does not prevent noise from corrupting it.
Why it matters: It separates the ability to send at a given rate from the ability to send reliably despite noise. - Two purposes of redundancy: Distinguish the naturally occurring redundancy removed during compression from the carefully designed redundancy added by channel coding, including their different effects on efficiency and reliability.
Why it matters: The apparent contradiction between removing and adding redundancy disappears once their different purposes are clear.
Key terms: noise, channel coding, structured redundancy, decoder, error-correcting codes, majority voting, noisy channel coding theorem, transmission rate, channel capacity, probability of error
5.2.4 Signal design
Signal design turns a compressed, error-protected message into physical signals while balancing how much information they carry against how reliably they can be distinguished in noise.
- Physical representation of messages: Explain why a compressed and error-protected message still needs a physical signal to travel through a channel, and give examples of signals used by different media.
Why it matters: It connects the information in a message to what actually travels through a communication channel. - Signal efficiency and reliability tradeoff: Explain how the number of distinguishable signal states affects both the information conveyed and susceptibility to noise, and why signal design balances the two.
Why it matters: The tradeoff explains why carrying more information is not the only aim of a communication system.
Key terms: physical signals, communication channel, digital modulation, distinguishable signal states, noise, channel's capacity
5.2.5 From communication to modern learning methods
The section explains why Shannon’s measurable view of information extends from communication systems to modern learning methods.
- Measurable Information Beyond Communication: Explain how defining information in terms of uncertainty made it useful beyond communication, and contrast the aims of communication systems with those of learning methods.
Why it matters: This connection explains why a theory developed for communication appears in statistical inference and machine learning.
Key terms: information, uncertainty, entropy, communication
5.3 From information theory to data partitioning
This introduction explains how Shannon’s measure of information extends from communication to learning from data.
- Information extracted from data: Explain how treating information as a measurable reduction in uncertainty lets learning algorithms use Shannon’s idea to learn from observations, not just analyze transmitted messages.
Why it matters: This connects Shannon’s communication theory to the way learning algorithms evaluate data.
Key terms: information, reduction of uncertainty, entropy, bits, observation
5.3.1 Decision trees
This section explains how decision trees use entropy and information gain to choose splits, illustrated with a loan example, and how those splits relate to variable importance.
- Decision tree structure and use: Describe how a decision tree makes predictions through feature-based decisions from a root node to a leaf, and distinguish its classification and regression uses.
Why it matters: The tree’s visible paths make its predictions relatively easy to trace. - Entropy guided split selection: Explain how entropy and information gain guide the choice among candidate splits and why the procedure repeats down the tree.
Why it matters: This connects Shannon’s measure of uncertainty to the tree’s concrete choice of questions. - Loan split information gain: Trace the loan example’s comparison of uncertainty before and after its proposed split, including how branch entropies are combined.
Why it matters: The example shows exactly what the tree compares when evaluating a proposed split. - Entropy and Gini criteria: Distinguish entropy from the Gini index as criteria for evaluating decision-tree splits.
Why it matters: It clarifies why a tree can pursue the same splitting objective without using entropy specifically. - Information gain and variable importance: Explain why predictors that repeatedly achieve large information gains receive higher variable-importance scores in a decision tree.
Why it matters: Importance scores complement the tree’s visible decision paths by comparing predictors’ contributions.
Key terms: root node, internal node, branch, leaf node, classification, regression, entropy, information gain, split, homogeneous groups
5.3.2 Random forests
This section explains how random forests combine diverse decision trees while using entropy reduction to choose splits and assess predictor importance.
- Random Forest Construction and Prediction: Explain why a single deep decision tree may overfit and how a random forest addresses that problem. Describe the two sources of variation among its trees and how their predictions are combined.
Why it matters: It connects the forest’s design to the problem of overfitting a single tree. - Entropy Guided Forest Splits: Describe how entropy and information gain guide the splits inside a random forest’s individual trees.
Why it matters: It shows that the forest’s randomness does not replace the information-based rule for choosing splits. - Entropy Based Predictor Importance: Explain how the section relates a predictor’s importance to reductions in entropy across the forest.
Why it matters: It extends the role of entropy from choosing splits to describing which predictors contribute most across the forest.
Key terms: overfit, bootstrapped sample, random subset of predictors, majority voting, averaging, entropy, uncertainty, information gain, candidate split, variable importance
5.3.3 Feature selection
This section shows how mutual information measures a predictor’s reduction of uncertainty about a target and guides feature selection.
- Purpose of feature selection: Explain why feature selection excludes predictors that contribute little useful information and how doing so can improve a model.
Why it matters: Feature selection helps avoid keeping predictors merely because they are available. - Mutual information calculation: Describe mutual information as the difference between a target’s entropy before a predictor is known and the uncertainty remaining after it is known. Trace how the patient example uses the blood-pressure groups to obtain that difference.
Why it matters: This calculation puts a quantity on how informative a predictor is about the outcome. - Predictor ranking versus tree splits: Distinguish using entropy reduction to evaluate a predictor for feature selection from using it to choose a decision-tree split, and apply the distinction to blood pressure and shoe size.
Why it matters: The same measure of information gain can answer different modeling questions.
Key terms: feature selection, predictors, target variable, mutual information, entropy, conditional entropy, bits, information gain, decision trees
5.3.4 Clustering
The section explains how entropy complements geometric clustering by measuring how much cluster membership reduces uncertainty about meaningful categories, and how information can also guide cluster formation.
- Unsupervised clustering and geometry: Describe how clustering finds groups in unlabeled data and how k-means and hierarchical clustering form groups using relationships among observations.
Why it matters: The way clusters are formed sets up the distinction between geometric fit and the information those clusters reveal. - Geometric versus informational evaluation: Distinguish geometric cluster evaluation from entropy-based evaluation, including when known categories are needed for the latter comparison.
Why it matters: Well-separated groups are not necessarily useful groups; their value can also depend on what they reveal. - Category agreement and cluster measures: Explain what cluster entropy, purity, and normalized mutual information indicate, and use the fruit example to compare pure and mixed clusters.
Why it matters: These measures make the difference between revealing categories and leaving them uncertain concrete. - Information as clustering objective: Distinguish using entropy to evaluate an existing clustering from using information-theoretic principles to seek the clusters themselves.
Why it matters: Information can be a criterion for forming groups, not only a score assigned afterward.
Key terms: clustering, unlabeled data, k-means clustering, centroid, hierarchical clustering, dendrogram, geometry, feature space, entropy-based evaluation, cluster membership
5.4 Information theory in deep learning
This section shows how Shannon’s ideas about information, uncertainty, and redundancy apply to neural-network training and representation learning.
- Cross-entropy in neural training: Explain that cross-entropy measures how closely a neural network’s predicted probabilities match true outcomes, and that training adjusts parameters to reduce this loss. State the section’s connection between minimizing cross-entropy and maximizing the likelihood of observed data.
Why it matters: It connects an information-theory measure to the objective a neural network reduces during training. - Compact informative representations: Explain why learning directly from high-dimensional, redundant raw data can be inefficient, and describe what representation learning aims to retain and remove.
Why it matters: It shows how information and redundancy guide what a learning system keeps from complex data.
Key terms: cross-entropy loss function, predicted probabilities, maximum likelihood estimation, representation learning, high-dimensional, redundancy, noise, downstream tasks
5.4.1 Neural networks
This section explains how cross-entropy turns a neural network’s probabilistic predictions into a measurable training objective, using digit recognition to illustrate uncertainty reduction.
- Cross-entropy as prediction measure: Explain how cross-entropy measures the discrepancy between a network’s predicted probabilities and the true outcomes, and why evaluating the whole distribution is more informative than checking whether the selected class is right.
Why it matters: It explains how a network can measure improvement even when its chosen class stays the same. - Incremental loss-guided learning: Trace how a network uses cross-entropy loss during training: after predictions, it makes small adjustments to weights and biases according to their contribution to the error, repeatedly shifting probability toward correct outcomes.
Why it matters: A measurable loss turns improving predictions into a process the network can repeatedly optimize. - Digit example of uncertainty: Describe the digit-classification example: an initially uniform prediction over ten digits has 3.32 bits of entropy, while a later prediction assigning 95% to the correct digit has about 0.44 bits. Explain what that reduction illustrates without treating the example’s prediction entropy as identical to cross-entropy loss.
Why it matters: The numbers make the section’s account of uncertainty reduction concrete.
Key terms: cross-entropy loss, predicted distribution, true distribution of outcomes, weights and biases, parameters, optimization, Shannon's entropy, bits, digit-classification network
5.4.2 Representation learning
This section explains representation learning as Shannon-inspired compression that removes redundancy while preserving information useful for a task.
- Purpose of learned representations: Explain that representation learning transforms raw, high-dimensional data into compact, structured descriptions useful for later tasks, rather than merely producing predictions. Distinguish the broader idea from neural networks, which commonly implement it.
Why it matters: It shifts attention from a network’s prediction to the useful form of data it learns along the way. - Information preserving compression: Explain how the section connects representation learning to Shannon’s source coding and entropy: predictable variation contributes little new information, but compression is valuable only if it retains information that matters.
Why it matters: Compression helps learning only when it removes what is expendable without losing what carries meaning. - Image encoder illustration: Describe the image example: an encoder replaces many predictable pixel values with a compact latent representation of informative structures, which can support recognition or a close reconstruction by a decoder.
Why it matters: The image makes the difference between keeping every detail and keeping useful structure concrete.
Key terms: representation learning, representations, raw data, source coding, entropy, redundancy, informative content, encoder, latent representation, decoder
5.5 From communication to learning
This section summarizes how Shannon’s measurement of uncertainty gave communication engineering a mathematical foundation and became useful for learning from data.
- Information and entropy in bits: Explain that Shannon defined information as a reduction of uncertainty and entropy as a measure of average uncertainty, making information, uncertainty, and redundancy quantifiable in bits.
Why it matters: Quantifying uncertainty is the link between Shannon’s communication work and its later uses in learning. - Communication limits under constraints: Describe what Shannon’s framework contributed to communication engineering: a rigorous way to establish limits for compression, transmission, and reliable communication amid bandwidth constraints and noise.
Why it matters: The original communication problem shows what Shannon’s measurements were first used to address. - Information measures in learning: Explain why Shannon’s ideas extend beyond communication: once information and uncertainty can be measured, they can be optimized in methods for learning from data.
Why it matters: This explains why an idea developed for communication remains relevant to machine learning.
Key terms: information, uncertainty, entropy, redundancy, bits, compression, transmission, reliable communication, bandwidth constraints, noise
5.6 Summary
Where this is used later
- Shannon: quantifying information and the limits of communication (chapter 10) · Revisits Shannon’s quantification of information and the limits it establishes.
- Message passing across codes and graphical models (chapter 10) · The discussion of message passing across codes builds on the purpose of error-correcting codes.
- The strengths and limitations of single decision trees (chapter 9) · The earlier introduction to decision trees as data partitions.
- Measuring variable importance (chapter 9) · Extends variable-importance measurement to random forests.
- The complete random forest algorithm (chapter 9) · The complete random forest algorithm develops this account of constructing and combining trees.
- (chapter intro) (chapter 9) · Introduces random forests in the context of data partitioning.
- Cross-entropy as likelihood and information (chapter 11) · Develops the connection between cross-entropy, likelihood, and neural-network learning.
- Learned representations simplify prediction (chapter 11) · Examines how learned representations simplify prediction.
Concepts explained in this chapter
- entropyEntropy is a quantitative measure of uncertainty across the possible outcomes of an information source and their probabilities.
- informationInformation is a quantifiable reduction in uncertainty about the state or outcome of a variable or system.
- redundancyRedundancy is the deliberate inclusion of repeated or structured information to enable reliable recovery of a message despite errors or noise.
- uncertaintyUncertainty is the lack of complete information about the state of a system, a quantity, or the outcome of an event.
- decision treeA decision tree is a hierarchical predictive model that recursively partitions data using feature-based rules.
- information gainInformation gain is the reduction in uncertainty produced by a split or observation, commonly measured using entropy.
- bitA bit is a binary unit of information that can take one of two possible values.
- cross entropy lossCross-entropy loss measures the discrepancy between target outcomes and predicted probabilities.
- clusteringClustering is an unsupervised modeling technique that groups observations according to similarity without predefined labels.
- decoderA communication-system component that reconstructs a transmitted message from its encoded form.
- encoderA communication-system component that converts a message into a form suitable for transmission.
- neural networkA neural network is a parameterized computational model composed of interconnected layers that transform inputs through learned functions.
- optimizationAn optimization method or algorithm is a procedure for searching for solutions that improve an objective function.
- parameterA parameter is a quantity that determines aspects of a model’s behavior.
- probability distributionA probability distribution assigns nonnegative probabilities to possible outcomes so that the probabilities over the defined outcome space sum to one.
- feature selectionFeature selection identifies a subset of predictors that contributes useful information while excluding predictors that add little value.
- cluster entropyCluster entropy measures the uncertainty in category membership among the observations within a cluster.
- normalized mutual informationNormalized mutual information is a normalized measure of the association between cluster assignments and known categories.
- purityPurity measures the extent to which each cluster is dominated by a single known category.
- data representationA representation encodes informative properties of data in a form suitable for processing or interpretation.
- representation learningRepresentation learning transforms raw data into representations that retain information useful for downstream tasks.
- mutual informationMutual information is a measure of the reduction in uncertainty about one variable provided by observing another variable.
- random forestA random forest is an ensemble model that combines predictions from multiple decision trees trained with randomized sampling or feature selection.
- channel capacityThe maximum rate at which information can be transmitted reliably through a communication channel under specified conditions.
- coding theoryCoding is the systematic representation of messages as symbols or codewords for efficient transmission, storage, or error protection.
- feature importance scoreVariable importance is a quantitative assessment of how much a predictor contributes to a model's predictions or reduction of uncertainty.
- Gini impurityGini impurity measures the probability of misclassifying an observation when its class is assigned randomly according to the class distribution in a node.
- independent variableA feature is an input attribute whose values are considered by a predictive model when making predictions.
- internal nodeAn internal node is a decision-tree node that tests a feature and directs observations along branches.
- leaf nodeA terminal node is a node marking the end of a decision-tree path and recording its resulting outcome.
- K-means clusteringK-means clustering partitions observations into a specified number of groups by assigning them to the nearest centroid and iteratively updating the centroids.
- hierarchical clusteringHierarchical clustering constructs nested groups by successively merging or splitting clusters.
- compressionEncoding information using fewer bits or symbols while preserving what is needed for its recovery or use.
- bootstrap sampleA dataset formed by drawing observations with replacement from an original dataset.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
