Active data collection chooses what to observe or ask next so the result reduces uncertainty as much as possible. Bayesian experimental design makes this choice by comparing expected information gain. section 13.7
The strategy proceeds by:
- Considering what each possible observation would imply about the unknown truth. section 13.7
- Quantifying uncertainty before and after each observation with entropy. section 13.7
- Selecting the observation with the lowest expected remaining uncertainty. section 13.7
Finding an unknown number
For an unknown integer from 1 to 100, asking whether it exceeds 50 splits the possibilities evenly and gains one bit of information. Repeating midpoint questions produces binary search, which identifies the number in at most seven questions and generally outperforms random guessing. section 13.7
def eig_for_threshold(x, N):
p_left = x / N #A
p_right = (N - x) / N #A
H_left = entropy(x) #B
H_right = entropy(N - x) #B
H_post = (
p_left * H_left
+ p_right * H_right
) #C
H_prior = entropy(N) #D
return H_prior - H_post #EA Python function computes expected information gain for a threshold by weighting the entropy of each possible outcome. section 13.7
