A point estimate gives one number for a quantity that can vary. In model validation, a single precision or recall score may look decisive while concealing uncertainty about how the model will perform on other data. Offline scores can also differ substantially from online performance. section 7.4
A coin toss, then a classifier
Count the heads in 100 flips of a fair coin: that count is a point estimate. Repeat the experiment and the count will likely change. A range, such as 40 to 60 heads in 95 out of 100 trials, expresses the variation the single count leaves out. Likewise, precision and recall varied across independently generated validation datasets even when the classifier, threshold, and data distribution stayed the same; substantial variation remained with 500,000 examples. section 7.4
Distribution recall with every point being a bootstrapped original dataset; the red line is a 99% lower confidence bound.
Bootstrapped recall values show a distribution rather than one score; the marked 99% lower confidence bound makes a downside threshold visible. section 7.4
A precise-looking number is not necessarily a useful decision criterion. On a system handling 100 billion events a day, a 0.1% departure from a point estimate means 100 million more misclassified events than expected. Uncertainty is only part of the check: the metric must also fit the decision. In fraud detection, recall calculated from known fraud cases missed the unknown cases, and a spam/nonspam metric did not fully capture whether users were experiencing too much spam. section 9.1.3
