A long message may reveal little if it only confirms what is already expected. A short message can convey much more if it resolves a highly uncertain outcome. In Shannon’s view, information is measured by the uncertainty it resolves, not by a message’s meaning or physical length. section 5.1 section 5.6
Entropy measures the average uncertainty associated with a source before a particular outcome is observed. The bit provides a scale for expressing that uncertainty. Predictable outcomes contribute less new information; surprising outcomes resolve more uncertainty. section 5.6 section 5.0
Selecting one of eight messages
If eight messages are equally likely, the selection has three bits of entropy. Learning which message was selected resolves that uncertainty. If some messages are more likely than others, entropy is lower because the outcome is more predictable. The three bits describe uncertainty, not the physical length of a message. section 10.2.4
In communication, information must reach a receiver despite constraints such as noise. Compression removes predictable redundancy; error-correcting codes add structured redundancy to protect a signal. A channel’s capacity limits the rate at which information can be conveyed reliably, even with coding. section 10.2.4 section 5.6
The same view connects information to learning: data provide evidence that can narrow uncertainty, while assumptions, representation, computation and data quality limit what can be learned. More data do not necessarily mean more useful information; repeated or formulaic examples may add little to the patterns already available. section 13.4.1 section 13.5.2
Information content is not the same as decision value. A test can reduce uncertainty yet have no value before costs if every possible result leaves the preferred action unchanged. Its net value is negative when the test also costs time or resources. section 7.1.3
Table 7.2 Information content compared with decision value
|
Framework
|
Core Question
|
Measure
|
Unit
|
Purpose
|
|
Claude Shannon (1948)
|
How much information does a message convey?
|
Entropy
|
Bits
|
Quantifies information and the limits of
efficient transmission
|
|
Howard Raiffa and Robert Schlaifer (1961)
|
How much does learning improve a decision?
|
Expected value of information
|
Utility units
|
Quantifies decision value
|
Compare information measured in bits with the expected value of information measured in utility units. section 7.1.3

