liveBook · Manning

Information theory

As explained in Timeless Algorithms, The Foundational Ideas

Information theory is the mathematical study of information, uncertainty, and communication, including measures of entropy and error.

Information theory offers a framework for reasoning under uncertainty, connecting Bayesian inference, statistical estimation, and machine learning. section 1.3

I=-log⁡(p)

An event assigned probability p has information cost I = -log(p); less probable events carry greater cost. section 11.4.2

In classification, this view interprets cross-entropy as information cost for observed outcomes. Minimizing binary cross-entropy is equivalent to maximizing the likelihood of the observed classes. section 11.4.2

Probability and information cost

Assigned probabilities of 0.68 and 0.99 yield information costs of 0.39 and 0.01, respectively. section 11.4.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines event information as negative log probability and relates binary cross-entropy to likelihood.
Examples
Illustrates event information with probabilities 0.68 and 0.99, yielding costs 0.39 and 0.01.
What you can do after
Understand why cross-entropy penalizes low probability assigned to observed classes and serves as a classification objective.
How it connects
Connects Bayesian inference, statistical estimation, machine learning, likelihood, and cross-entropy.

Also explained in

Compare how 3 books explain information theory →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.