liveBook · Manning

Information

As explained in Timeless Algorithms, The Foundational Ideas

Information is a quantifiable reduction in uncertainty about the state or outcome of a variable or system.

A long message may reveal little if it only confirms what is already expected. A short message can convey much more if it resolves a highly uncertain outcome. In Shannon’s view, information is measured by the uncertainty it resolves, not by a message’s meaning or physical length. section 5.1 section 5.6

Entropy measures the average uncertainty associated with a source before a particular outcome is observed. The bit provides a scale for expressing that uncertainty. Predictable outcomes contribute less new information; surprising outcomes resolve more uncertainty. section 5.6 section 5.0

Selecting one of eight messages

If eight messages are equally likely, the selection has three bits of entropy. Learning which message was selected resolves that uncertainty. If some messages are more likely than others, entropy is lower because the outcome is more predictable. The three bits describe uncertainty, not the physical length of a message. section 10.2.4

In communication, information must reach a receiver despite constraints such as noise. Compression removes predictable redundancy; error-correcting codes add structured redundancy to protect a signal. A channel’s capacity limits the rate at which information can be conveyed reliably, even with coding. section 10.2.4 section 5.6

The same view connects information to learning: data provide evidence that can narrow uncertainty, while assumptions, representation, computation and data quality limit what can be learned. More data do not necessarily mean more useful information; repeated or formulaic examples may add little to the patterns already available. section 13.4.1 section 13.5.2

Information content is not the same as decision value. A test can reduce uncertainty yet have no value before costs if every possible result leaves the preferred action unchanged. Its net value is negative when the test also costs time or resources. section 7.1.3

Table 7.2 Information content compared with decision value
Framework
Core Question
Measure
Unit
Purpose
Claude Shannon (1948)
How much information does a message convey?
Entropy
Bits
Quantifies information and the limits of efficient transmission
Howard Raiffa and Robert Schlaifer (1961)
How much does learning improve a decision?
Expected value of information
Utility units
Quantifies decision value

Compare information measured in bits with the expected value of information measured in utility units. section 7.1.3

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains uncertainty and entropy through a quantitative message-selection example and distinguishes information content from decision value.
Examples
Uses eight equally likely messages and neural language-model prediction to illustrate information and uncertainty.
What you can do after
Evaluate how uncertainty reduction differs from the expected value of information in a decision.
How it connects
Connects information to entropy, compression, channel capacity and decision trees.

Also explained in

Compare how 2 books explain information →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.