A functional problem occurs when a system stops providing an expected capability because changing entities, pressures, technologies, or behaviors make its learned relationships irrelevant. Unlike ordinary software defects, ML systems can fail functionally even when their implementation remains unchanged. section 9.5
Manage functional problems as production-observability concerns:
- Log events relevant to operation, including database connections, user logins, data updates, and model decisions. section 9.5
- Generate alarms for functional behavior as well as nonfunctional behavior, so emerging failure can be located and timed. section 9.5
- Set logging density and retention from application requirements while balancing monitoring load and storage cost. section 9.5
Changing music markets
A model predicting music sales from pre-streaming data may stop working after streaming becomes widespread, even without a software bug. section 9.5
