Bootstrapped lower confidence bound of recall at a given precision

As explained in Machine Learning System Design

A one-sided bootstrap-based lower confidence estimate of recall conditional on achieving a specified precision.

This metric gives a one-sided, bootstrap-based lower confidence estimate of recall while requiring the model to achieve a specified precision. It is designed to make performance estimates more reliable and robust. section 7.4

The estimation combines:

• Poisson bootstrap resampling to show the distribution of recall and precision. section 7.4
• Noise addition to test whether the decision boundary and resulting metric remain robust. section 7.4

Robustness check

When added noise increases recall and decreases precision, the differing distributions indicate a narrow, nonrobust decision-boundary margin. The lower bound therefore communicates performance more cautiously than a single recall value. section 7.4

Distribution recall with every point being a bootstrapped original dataset; the red line is a 99% lower confidence bound.

Inspect the recall distribution and its red lower-confidence-bound line. section 7.4

How Machine Learning System Design covers it

How deep it goes

Combines Poisson bootstrap, noise perturbation, and confidence bounds, stopping short of a mathematical derivation.

Examples

Illustrates recall distributions, 99% lower bounds, bootstrap samples, and added-noise comparisons.

What you can do after

Evaluate model performance using a conservative and robustness-aware recall estimate.

How it connects

Connects recall with precision, specificity, decision-boundary robustness, and confidence intervals.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.