Training-serving skew occurs when a model learns from artificially constructed, cleaned, incomplete, or otherwise unrepresentative data, then encounters substantially different data in production. The mismatch can reduce real-world performance even when offline results look strong. section 14.3.5
Cloudy development, sunny deployment
An augmented-reality team collected and tested camera data in cloudy, rainy St. Petersburg. Investors later tested the product in sunny Cyprus, where the computer-vision component performed poorly because the deployment environment differed from the training environment. section 14.3.5
Reduce the risk by aligning data work with the real application:
- Collect and label data from unsuccessful trials, then retrain or adapt the existing model when those examples expose the mismatch. section 14.3.5
- Avoid filtering genuine user inputs too aggressively; unusual queries may be messy, but removing them can make training data unlike production traffic. section 14.5.2
- Compare the diversity and quality of training examples with real images, queries, labels, and emerging user or content patterns. section 14.3.5 section 14.5.2
Figure 14.5 Such a drastic difference between sandbox sets of images and real-life photos can lead to significant drops in the model’s performance.
Use the figure to contrast controlled image collections with varied real-life photos and identify visual sources of performance loss. section 14.3.5
