Information theory

As explained in Bayesian Optimization in Action

Information theory is the mathematical study of information, uncertainty, and the relationships between probability distributions and observations.

Information theory gives a mathematical way to represent, quantify, and reason about information under uncertainty. In decision-making, it asks which action is expected to leave the least uncertainty about the quantity that matters. This turns information gathering into a search objective: prefer observations that most reduce uncertainty. section 6.1 section 6.0

Entropy measures uncertainty in a probability distribution. For possible events with probabilities pᵢ, entropy is −Σᵢ pᵢ log pᵢ. A uniform belief over many possibilities has high entropy because every outcome remains plausible; a posterior concentrated on fewer possibilities has lower entropy. Information gain is the reduction in entropy after an observation. section 6.1.3 section 6.1.2

Searching for a remote

Suppose the remote is believed to be uniformly located somewhere in the living room. Searching part of that room reduces entropy in either outcome: finding it leaves no location uncertainty, while not finding it restricts the posterior to the unsearched region. Searching the bathroom is much less useful because it is unlikely under the prior belief and usually leaves that belief unchanged. section 6.1.2

Figure 6.3 Entropy of the location of the remote after a portion of the living room is searched. If the remote is found (upper right), then no uncertainty remains. Otherwise, entropy is still reduced (lower right), as the distribution of the location of the remote is now narrower.

The floor-plan outcomes make entropy reduction visible: a successful search eliminates uncertainty, while an unsuccessful but relevant search narrows the possible region. section 6.1.2

Binary search illustrates the same principle. If a target is equally likely to occupy any position in a sorted 100-element list, inspecting the middle divides the possibilities into two equal halves. After the comparison, one half can be discarded, producing a lower-entropy posterior. The middle position minimizes expected posterior entropy, which explains why binary search beats checking positions sequentially on average. section 6.1.3

Figure 6.9 Expected posterior entropy in the location of the target as a function of the location within the list to inspect. The middle location is optimal, minimizing the expected entropy.

The expected-posterior-entropy curve shows why the middle comparison is optimal: it leaves the least uncertainty on average. section 6.1.3

The prior distribution matters. When possible answers are not equally likely, an entropy-guided action should account for that asymmetry and evaluate how each outcome updates the posterior. For example, phone-drop thresholds can begin with an exponential prior over the highest safe floor, then be narrowed after each break or non-break result. section 6.3.1

In Bayesian optimization, entropy supplies a policy heuristic distinct from improvement-based, upper-confidence-bound, and Thompson-sampling policies. Rather than directly seeking a high observed value, an entropy-based policy chooses evaluations expected to increase information about the objective function or its global optimum, balancing exploration and exploitation through uncertainty reduction. section 6.0

How Bayesian Optimization in Action covers it

How deep it goes

Defines entropy, computes it for a uniform search distribution, and derives binary search through expected posterior entropy.

Examples

Uses remote-control search, a 100-element binary search, and a prior-informed phone-drop threshold exercise.

What you can do after

Apply it to choose searches or evaluations that minimize expected posterior uncertainty under a specified prior.

How it connects

Connects entropy and information gain to Bayesian optimization, binary search, posterior distributions, exploration, and exploitation.

liveBookmanning.com

Read Bayesian Optimization in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  How Large Language Models WorkThe mathematical study of information as a quantifiable resource, including its representation, transmission, and uncertainty.

Compare how 2 books explain information theory →

Related concepts

• informationmeasures · Information is content that conveys meaningful distinctions or reduces uncertainty about possible states, observations, or outcomes.
• context assemblyuses · Context assembly is the process of selecting, organizing, and combining relevant information and instructions into context for a model or other downstream processing.
• entropyuses · Entropy is a quantitative measure of uncertainty or dispersion in a probability distribution and, more generally, of disorder in the possible states of a system.
• binary searchuses · Binary search is an algorithm that locates a target or boundary in an ordered collection by repeatedly comparing with a midpoint and halving the remaining search range.
• Bayesian optimizationuses · Bayesian optimization is a sequential method for optimizing an expensive or unknown objective by using a probabilistic model to guide evaluations.
• max-value entropy searchused by · Max-value Entropy Search is a Bayesian optimization acquisition policy that selects evaluations expected to reduce uncertainty about the global maximum value of an objective.

This explainer was generated by Manning's knowledge graph from the text of Bayesian Optimization in Action. The section links point to the parts of the book it draws on.