A classifier may output a probability, but an application often needs a label. A decision threshold is the cutoff used to assign that label. If data drift changes the data the model encounters, changing the threshold is one possible response without changing the model itself. section 14.4.4
A significance level is a different kind of threshold. In a statistical test, a result is called significant when its p-value is below the chosen significance level, α. A common choice is α = 0.05, corresponding to a 5% type I error threshold. This cutoff judges evidence for an experimental effect; it does not assign a model prediction to a class. section 12.2.4 section 12.2.1
Choose the cutoff for the decision it actually controls:
- For classification, consider a threshold adjustment when data drift is detected. Treat it as an application-level change that needs care: added business logic can have unintended consequences. section 14.4.4
- For an A/B test, set the significance level during experiment design, alongside the type II error threshold and sample-size planning. Check planned error rates with simulations on historical data; do not change the experiment design on the go. section 12.2.1
A retail experiment
A retail evaluation uses Welch’s t-test, a 5% significance level, and a 10% type II error threshold. It extends the planned two-week test to a full month to account for the distribution centers’ weekly replenishment cadence. These thresholds govern how the experiment is assessed, not how a classifier labels individual predictions. section 12.4.1
