Offline evaluation estimates the effect of a model change using pre-collected data, without exposing its outputs to users in a live product. It gives teams a fast feedback loop for finding weaknesses and comparing versions before deciding whether a change is ready for online testing. section 1.3.1 section 1.2
Use two complementary layers. A canonical evaluation compares versions on a fixed dataset, consistent setup, and defined metrics. A diagnostic evaluation examines behavior across slices and scenarios: for example, whether recommendations differ for new and active users, or whether exposure is concentrated on a narrow range of content. An aggregate improvement can hide these problems. section 1.4
A practical evaluation loop:
- Choose metrics that reflect the product goal. Ranking, classification, and task completion call for different measures; a scoring method such as an LLM judge is not itself the quality dimension being measured. section 1.3.4
- Run candidate versions against a consistent dataset and setup. Compare results, inspect important user and content slices, and investigate regressions before promoting a candidate. section 1.4 section 1.2
- Iterate when results are negative or inconclusive. Promising results can justify an online experiment, but do not establish real user impact on their own. section 1.3 section 1.6
- After launch, recompute suitable metrics on fresh production logs to detect regressions or drift. Some signals require delayed labels or review, so they may not be available immediately. section 1.7.1
Evaluating streaming recommendations
For a movie recommendation row, evaluate whether the top-ranked titles are relevant using Precision@K. For a row meant to help users find their recent shows, recall emphasizes retrieving the relevant past shows. Inspect slices as well: a better aggregate score may coexist with weaker recommendations for new users or concentrated exposure for a small set of titles. section 1.3 section 1.4
Static data cannot fully capture how a model changes future user behavior. It can also miss interaction details, such as an accurate voice response that is too long to be useful. Supplement offline results with online experiments, user studies, or simulations where those effects matter. For internal tools, historical ticket comparisons or replayed cases may provide useful evaluation even when a traditional A/B test is not used. section 1.8.1 section 1.8.3 section 1.3.2
Treat offline metrics as signals, not guarantees. Compare them with later online outcomes to learn how predictive they are, and validate and calibrate any LLM judge used for scoring. Product, engineering, and data science may need to work together to define quality, run the evaluation, and interpret its results. section 1.7.2 section 1.3.4 section 1.8.4




