Regression is a defect introduced by a change that makes previously working behavior fail or deteriorate. In machine-learning systems, a model update can improve some test cases while creating new harmful errors elsewhere, so a better aggregate score does not automatically make the release safe. section 13.2
Treat regression as a release decision, not only a test result:
- Run representative tests and review software-related coverage, while recognizing that tests alone cannot guarantee that a model change avoids harmful outcomes. section 13.2
- Compare improvements against newly introduced errors instead of relying only on overall performance. section 13.2
- Use human review when the tradeoff is difficult to judge; responsibility may belong to an ML engineer, product manager, domain expert, or aggregated user-feedback service. section 13.2
- Test training pipelines to catch regression bugs earlier, increase iteration speed, and expose poor configurability choices. section 10.7
A model that scores better but releases a regression
A final test set contains 100 samples. A new model corrects three previous errors but introduces two new errors elsewhere. Although its overall performance improves, release approval still requires judging whether the new failures are acceptable. section 13.2
