liveBook · Manning

Information theory

Information theory is the mathematical study of information, uncertainty, and the quantification, encoding, storage, processing, and communication of information.

How 3 Manning books explain it

Bayesian Optimization in Action

Information theory is the mathematical study of information, uncertainty, and the relationships between probability distributions and observations.

How it is used
Guides Bayesian optimization decisions toward evaluations expected to reduce uncertainty about an objective function or global optimum.
How deep it goes
Defines entropy, computes it for a uniform search distribution, and derives binary search through expected posterior entropy.
Examples
Uses remote-control search, a 100-element binary search, and a prior-informed phone-drop threshold exercise.
What you can do after
Apply it to choose searches or evaluations that minimize expected posterior uncertainty under a specified prior.

Read how Bayesian Optimization in Action explains information theory →

How Large Language Models Work

The mathematical study of information as a quantifiable resource, including its representation, transmission, and uncertainty.

How it is used
Uses information as a measurable resource to assess claims that language models can improve themselves through repeated self-training.
How deep it goes
Defines information distribution and noisy reproduction, but stops short of mathematical derivations or formal measures.
Examples
Uses an iterative loop of generating data, retraining, and observing degraded quality to illustrate information loss.
What you can do after
Evaluate/understand why synthetic data alone cannot supply new information for indefinite model improvement.

Read how How Large Language Models Work explains information theory →

Timeless Algorithms, The Foundational Ideas

Information theory is the mathematical study of information, uncertainty, and communication, including measures of entropy and error.

How it is used
Interprets cross-entropy as an information cost for observed outcomes in classification.
How deep it goes
Defines event information as negative log probability and relates binary cross-entropy to likelihood.
Examples
Illustrates event information with probabilities 0.68 and 0.99, yielding costs 0.39 and 0.01.
What you can do after
Understand why cross-entropy penalizes low probability assigned to observed classes and serves as a classification objective.

Read how Timeless Algorithms, The Foundational Ideas explains information theory →

Also covered in

What to know first

Related concepts

Questions about information theory

What is information theory?

Information theory is the mathematical study of information, uncertainty, and the quantification, encoding, storage, processing, and communication of information.

Which Manning books explain information theory?

Bayesian Optimization in Action, How Large Language Models Work and Timeless Algorithms, The Foundational Ideas.

What should I know before learning information theory?

Manning books that teach information theory usually introduce objective function first.

What are the parts of information theory?

Information theory includes entropy.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.