An invariant is a property that should remain unchanged under a specified transformation, perturbation, or model change. In ML products, invariants turn expectations such as “small input changes should not alter the decision” into measurable consistency requirements. section 5.2.1
Useful invariants depend on the system:
- A vision model should usually preserve its output after minor lighting changes, while a sentiment model should tolerate replacing words with synonyms. section 5.2.1
- A model should produce the same output for unchanged input, even across repeated executions or retraining with additional data or different seeds. section 5.2.1
- Property-based tests can generate inputs and verify invariants such as transformation invariance, robustness to small perturbations, and monotonicity. section 10.5.1
Antifraud release check
If the same user is classified as fraudulent today, legitimate tomorrow, and fraudulent again next week without a meaningful input change, the system violates a practical invariant. Measuring consistency across model releases helps prevent unexpected downstream effects. section 5.2.1
Figure 5.5 New model releases are fairly consistent when estimating the probability (P) of the user (U) being fraudulent (F).
Use the release-comparison figure to inspect how consistently successive models estimate fraud probability for the same user. section 5.2.1
