A prediction pipeline can be technically healthy and still produce faulty results when its inputs are incomplete, duplicated, malformed, or unexpectedly changed. Data quality monitoring checks whether training and prediction data meet agreed expectations, protecting model reliability and user trust. When quality falls too far, pause the model, use a fallback, or investigate and resolve the cause. section 14.3 section 14.3.6
Build checks around the inputs that can change model behavior:
- Check missingness for the whole input set and for individual features, including alternate markers such as “N/A,” “NaN,” and “undefined.” Set acceptable ranges and stricter policies for important features. section 14.3.6
- Check for duplicated data, which can distort inputs even when no records appear to be missing. section 14.3.6
- Check model inputs at a granular level rather than relying only on warehouse-wide monitoring. Add checks for feature transformations, real-time inputs, and external sources where upstream controls are insufficient. section 14.3.6
- Watch for schema changes, especially category changes, business-system updates, and newly introduced APIs or data sources that the model has not learned to interpret. section 14.3.4
A catalog change breaks expectations
Suppose a demand-forecasting or e-commerce recommendation system receives an updated product catalog. New or changed categories can alter the model’s interpretation of features. Schema-change alerts can expose the issue; the team can then investigate, educate business users about the consequences, and determine whether retraining or a fallback is needed. section 14.3.4 section 14.3
Monitoring is most useful when thresholds have an operational response. Visual checks can help establish missingness limits, while alerts can support batch jobs alongside success rate, failure rate, processed rows, prediction accuracy, and resource metrics. section 14.3.6 section 15.5.1
