liveBook · Manning

False positive

As explained in Experimentation for Engineers

A false positive is an incorrect conclusion that an observed effect or condition is present when it is not.

A false positive is an A/B-test decision that treats a variant as better when it is not. In a trading-routing decision, it means switching from ASDAQ to BYSE even though BYSE does not truly have lower expected costs. It is a controlled risk, not proof: a result beyond the significance threshold triggers an action while retaining a stated chance of being wrong. section 2.3.1 section 2.3.3

The false-positive rate is set by the decision threshold. With a unit-normal z score and a one-sided threshold of z < -1.64, 5% of measurements fall in the action region under the no-improvement assumption. Moving the threshold right raises false positives; moving it left lowers them. The normal approximation improves as the aggregate contains more individual measurements. section 2.3.1

Design and run tests so the stated error rate remains meaningful:

  • Choose a practically significant cost difference and estimate per-measurement variation, potentially from a pilot study. section 2.3.2
  • Set the required sample size before measuring; seven measurements in the illustrated design limit false positives to 5%, while adding a 20% false-negative limit requires more observations. section 2.3.2
  • Randomize each incoming order between alternatives and collect at least the designed number of measurements to reduce bias. section 2.3.3
  • Prespecify the business metric. Selecting a favorable metric after examining many metrics turns a single-metric 5% rate into a much larger family-wise risk. section 8.3.1

Checking a lower-cost routing result

Compute the aggregate cost difference, delta = mean(BYSE) − mean(ASDAQ), then divide by its standard error to obtain z. An observed delta of about −2.7 mips is practically significant when the target is 1.0 mip; z ≈ −6.35 also passes the −1.64 statistical threshold. Routing all trades to BYSE is then justified as a decision, while acknowledging the designed 5% false-positive risk. section 2.3.3 section 2.3.1

Figure 8.4 The probability of any one of a family of metrics generating a false positive versus the number of metrics, M, in the family
08-04

Compare the probability that at least one metric produces a false positive as the number of examined metrics grows. section 8.3.1

How Experimentation for Engineers covers it

How deep it goes
Defines threshold-based error, derives its 5% rate, and develops sample-size and multiple-metric consequences.
Examples
Uses ASDAQ-versus-BYSE execution costs and a model evaluated across several business metrics.
What it assumes
Uses aggregate measurements, standard errors, and z scores while explaining the normal approximation and threshold choice.
What you can do after
Evaluate/understand whether a significant A/B-test result may be a controlled mistaken action.
How it connects
Connects A/B testing with statistical significance, z scores, false negatives, power analysis, randomization, and multiple comparisons.

Also explained in

Compare how 26 books explain false positive →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Experimentation for Engineers. The section links point to the parts of the book it draws on.