Information theory offers a framework for reasoning under uncertainty, connecting Bayesian inference, statistical estimation, and machine learning. section 1.3
An event assigned probability p has information cost I = -log(p); less probable events carry greater cost. section 11.4.2
In classification, this view interprets cross-entropy as information cost for observed outcomes. Minimizing binary cross-entropy is equivalent to maximizing the likelihood of the observed classes. section 11.4.2
Probability and information cost
Assigned probabilities of 0.68 and 0.99 yield information costs of 0.39 and 0.01, respectively. section 11.4.2


