Suppose a doctor can order several tests, but each costs time or money. Which test should come next? Bayesian experimental design treats that choice as a way to learn about an unknown quantity, such as a patient’s condition. Rather than collecting data indiscriminately, it chooses an observation expected to reduce uncertainty the most. section 13.0 section 13.1
Entropy measures uncertainty from the probabilities assigned to possible outcomes. For each candidate experiment, expected information gain (EIG) is the current entropy minus the average entropy expected after observing its result. A larger EIG means a more informative choice on average—not a guarantee that every possible result will be equally helpful. section 13.11 section 13.7
The choice becomes a repeating procedure:
- Start with a prior belief about the unknown quantity and propose possible experiments. section 13.8
- Estimate each experiment’s EIG, then run the one with the highest value. section 13.8
- Observe the result, update the belief, and use it to choose the next experiment. section 13.8
Find a number from 1 to 100
Assume every number is initially equally likely. You may ask, “Is it greater than x?” Asking about 50 splits the possibilities evenly. Initial entropy is about 6.64 bits; either answer leaves 50 possibilities and about 5.64 bits, so EIG is 1 bit. An uneven split gives less information on average. In Python, math.log2(num_states) calculates entropy for a uniform set; a threshold’s EIG is that initial entropy minus the probability-weighted entropy of its two resulting sets. Repeat with the remaining numbers until one remains. section 13.5 section 13.7
Figure 13.13 The general framework of Bayesian experimental design: we choose the experiment that yields the most information about the quantity of interest on average, which updates our belief about that quantity and informs our next experiment.

The diagram shows how selecting an experiment, observing its outcome, and updating a belief form a feedback loop. section 13.8
The same principle appears in search and rescue, scientific measurement, and active learning, where collecting every possible observation can be costly. Maximizing EIG is also a decision-under-uncertainty problem: reducing entropy serves as the utility of an experiment. section 13.1 section 13.7
