Out-of-distribution (OOD) data differs meaningfully from the data used to train or evaluate a model. It is a distribution shift, not merely a rare example from the same long tail. section 3.2.3
Use OOD performance to locate production risks:
- Compare quality on familiar, in-distribution data with quality on OOD or stress-test data. section 3.2.3
- Investigate poor OOD results through representative training data, metadata coverage, fallback logic, or less brittle behavioral signals. section 3.2.3
Movie recommendation
Test users from a new geography, movies from a newly added content vertical, new metadata formats, or behavior shifted by a product launch. Compare recommendation quality with familiar users to identify where production performance may fail. section 3.2.3
The same diagnostic applies to language models and agents: vary wording, typos, dialects, irrelevant context, failed tools, incomplete APIs, ambiguous instructions, or unusually long tasks. section 3.2.3


